00010000:00010000:3.0:1787826287.458726:0:224016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458745:0:224016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458757:0:224016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458769:0:224016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458782:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458791:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458799:0:224016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826287.458811:0:224016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223480 x1874663332542208/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826287.458826:0:224016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458839:0:224016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826287.458845:0:224016:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826287.458873:0:224016:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826287.458880:0:224016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608010b3 00000080:00010000:3.0:1787826287.458886:0:224016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826287.458889:0:224016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608010b3 from it ffffb512c6d6bbd0 00010000:00010000:3.0:1787826287.458892:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.458902:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.458925:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826287.458944:0:224016:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=24 00000080:00200000:3.0:1787826287.458948:0:224016:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826287.458968:0:224016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826287.459009:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.459017:0:224016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.459025:0:224016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826287.459044:0:224016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542336:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826287.459877:0:224016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826287.474596:0:224016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542336:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826287.474622:0:224016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474648:0:224016:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474654:0:224016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474675:0:224016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474689:0:224016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474704:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474712:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474719:0:224016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826287.474731:0:224016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb223480 x1874663332542336/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826287.474744:0:224016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474758:0:224016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7748 (0 0) 00000002:00010000:3.0:1787826287.474775:0:224016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e44:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826287.474819:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e44:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474827:0:224016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.474836:0:224016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826287.474866:0:224016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542464:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826287.474972:0:224016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826287.489365:0:224016:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057c7539000: committing for last_committed 4294985347 gen 1 00000100:00100000:3.0:1787826287.489574:0:224016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542464:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826287.489594:0:224016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489610:0:224016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489631:0:224016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489643:0:224016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489673:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489681:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489689:0:224016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826287.489701:0:224016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb221180 x1874663332542464/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826287.489716:0:224016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489730:0:224016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7748 (0 0) 00000080:00200000:3.0:1787826287.489743:0:224016:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826287.489776:0:224016:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e44:0x0], updating i_size 4096 00000080:00200000:3.0:1787826287.489779:0:224016:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e44:0x0](ffff9057dfddba88): rc = 0 00000080:00010000:3.0:1787826287.489788:0:224016:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e44:0x0](ffff9057dfddba88) for remote lock 0x95194e87608010c1 00000080:00010000:3.0:1787826287.489793:0:224016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e44:0x0](ffff9057dfddba88) for lock 0x95194e87608010ba 00000080:00010000:3.0:1787826287.489803:0:224016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e44:0x0](ffff9057dfddba88) 00000080:00010000:3.0:1787826287.489806:0:224016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608010ba from it ffffb512c6d6baa0 00010000:00010000:3.0:1787826287.489810:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489819:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826287.489832:0:224016:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e87608010c1 from it ffffb512c6d6baa0 00010000:00010000:3.0:1787826287.489835:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.489841:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608010c1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c3f expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.489858:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e44:0x0](ffff9057dfddba88), inode mode 41ed mask 1 00000080:00200000:3.0:1787826287.489879:0:224016:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826287.489893:0:224016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826287.489903:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.489913:0:224016:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.489918:0:224016:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826287.489922:0:224016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826287.490063:0:224016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608010b3 00000080:00010000:3.0:1787826287.490069:0:224016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608010b3 from it ffffb512c6d6bbd0 00010000:00010000:3.0:1787826287.490072:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.490080:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826287.490090:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826287.490096:0:224016:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=24 00000080:00200000:3.0:1787826287.490101:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e44:0x0](ffff9057dfddba88), inode mode 41ed mask 1 00000080:00200000:3.0:1787826287.490115:0:224016:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88), flags=0 00000080:00200000:3.0:1787826287.490119:0:224016:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88), intent=getattr 00000002:00010000:3.0:1787826287.490131:0:224016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x0:0x0:0x0]) in obj [0x240000400:0x190:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826287.490172:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x190:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.490191:0:224016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x190:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.490200:0:224016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x190:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826287.490535:0:224016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542592:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826287.490743:0:224016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826287.501882:0:224016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542592:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826287.501913:0:224016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x190:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502005:0:224016:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502014:0:224016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502024:0:224016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502039:0:224016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502052:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502061:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502068:0:224016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826287.502081:0:224016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221180 x1874663332542592/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826287.502092:0:224016:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826287.502106:0:224016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502117:0:224016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7749 (0 0) 00000080:00200000:3.0:1787826287.502124:0:224016:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x190:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826287.502593:0:224016:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e45:0x0](ffff9057dfdeaa08): rc = 0 00000080:00200000:3.0:1787826287.502615:0:224016:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e45:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826287.502631:0:224016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e45:0x0](ffff9057dfdeaa08) for lock 0x95194e87608010c8 00000080:00010000:3.0:1787826287.502641:0:224016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e45:0x0](ffff9057dfdeaa08) 00000080:00010000:3.0:1787826287.502645:0:224016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608010c8 from it ffffb512c6d6bdb8 00010000:00010000:3.0:1787826287.502649:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502667:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.502703:0:224016:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88), flags=2048 00000080:00200000:3.0:1787826287.502707:0:224016:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88), intent=getattr 00000002:00010000:3.0:1787826287.502726:0:224016:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x18e:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826287.502768:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502777:0:224016:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.502785:0:224016:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224016 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826287.502804:0:224016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542720:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826287.502942:0:224016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826287.514252:0:224016:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332542720:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826287.514280:0:224016:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: --/CR res: [0x200000401:0x18e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514321:0:224016:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514344:0:224016:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514355:0:224016:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514366:0:224016:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514555:0:224016:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514565:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514574:0:224016:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826287.514588:0:224016:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221180 x1874663332542720/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826303 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826287.514600:0:224016:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826287.514614:0:224016:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514714:0:224016:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7606 (0 0) 00000080:00200000:3.0:1787826287.514722:0:224016:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x18e:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826287.514772:0:224016:0:(namei.c:147:ll_iget()) got inode: [0x200000403:0x1db6:0x0](ffff9057dfdeec08): rc = 0 00000080:00200000:3.0:1787826287.514786:0:224016:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1db6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826287.514791:0:224016:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1db6:0x0](ffff9057dfdeec08) for lock 0x95194e87608010cf 00000080:00010000:3.0:1787826287.514801:0:224016:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1db6:0x0](ffff9057dfdeec08) 00000080:00010000:3.0:1787826287.514804:0:224016:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608010cf from it ffffb512c6d6bdb8 00010000:00010000:3.0:1787826287.514808:0:224016:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826287.514817:0:224016:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.514847:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e44:0x0](ffff9057dfddba88), inode mode 41ed mask 3 00000080:00200000:3.0:1787826287.514853:0:224016:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e44:0x0](ffff9057dfddba88), inode mode 41ed mask 3 00000080:00200000:3.0:1787826287.514862:0:224016:0:(namei.c:2492:ll_rename()) VFS Op:oldname=f45j.sanityn-2, src_dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88), newname=f45j.sanityn, tgt_dir=[0x240000403:0x1e44:0x0](ffff9057dfddba88) 00010000:00010000:3.0:1787826287.514903:0:224016:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.514917:0:224016:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e45:0x0]: losing layout lock 00010000:00010000:3.0:1787826287.514960:0:224016:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.514967:0:224016:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826287.514996:0:224016:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e87608010c8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e45:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f49c5b expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.515027:0:224016:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826287.515037:0:224016:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1db6:0x0]: losing layout lock 00010000:00010000:3.0:1787826287.515722:0:224016:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.515731:0:224016:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826287.515734:0:224016:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608010cf lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1db6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f49c77 expref: -99 pid: 224016 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1787826287.515761:0:224016:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826287.515770:0:224016:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb223480 x1874663332543104/t0(0) o36->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:13/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826287.515792:0:224016:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:224016:1874663332543104:192.168.203.156@tcp:36:mrename.0 00000100:00100000:3.0:1787826287.515866:0:224016:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826287.520912:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332542848:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826287.552963:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346139136 00000100:00100000:3.0:1787826287.552973:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346139136 00000100:00100000:3.0:1787826287.553027:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43507 00000100:00100000:3.0:1787826287.553038:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346139136:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826287.553064:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e876080109e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49be4 expref: -99 pid: 224014 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.553131:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826287.553138:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346139136:12345-192.168.203.156@tcp:104: Request processed in 103us (424us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826287.553145:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43507 00000100:00100000:3.0:1787826287.563958:0:6159:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826287.563979:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346139392 00000100:00100000:3.0:1787826287.563988:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346139392 00000100:00100000:3.0:1787826287.564041:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43508 00000100:00100000:3.0:1787826287.564061:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c6d8bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:24218:x1874663346139392:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826287.564077:0:6159:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e8760801074 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x51d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b6d expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826287.564090:0:6159:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e8760801074 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x51d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b6d expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826287.564118:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@000000009ade4c10[0x110, 2, [0x200000403:0x1db6:0x0] hash] 00000020:00010000:3.0:1787826287.564120:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826283 atime: 1787826283 ctime: 1787826283 blocks: 1 00000020:00010000:3.0:1787826287.564124:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@000000008a5848c0[0x110, 3, [0x280000401:0x51d:0x0] hash] 00000020:00010000:3.0:1787826287.564125:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826283 atime: 1787826283 ctime: 1787826283 blocks: 1 00000100:00100000:3.0:1787826287.564190:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c6d8bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:24218:x1874663346139392:12345-192.168.203.156@tcp:106: Request processed in 139us (1076us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826287.564197:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43508 00000100:00100000:3.0:1787826287.569075:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826287.585838:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826288.037324:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332544768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826288.042642:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332544768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826288.442639:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:3.0:1787826288.546711:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826288.546734:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826288.546765:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826288.546789:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826288.546793:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826288.546818:0:192756:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826288.546841:0:192756:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826288.546926:0:192756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826288.547660:0:192756:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb223100 - object 0x2c0000400:1224 offset 0<>6 00000100:00100000:3.0:1787826288.547771:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332545280:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826288.553600:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826288.553647:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332545152:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826288.559752:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332545280:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826288.562968:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826288.563018:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826288.563058:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332545152:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826289.101517:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346145024 00000100:00100000:3.0:1787826289.101527:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346145024 00000100:00100000:3.0:1787826289.101586:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43514 00000100:00100000:3.0:1787826289.101597:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346145024:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826289.101615:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608010f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18e:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49d18 expref: -99 pid: 224052 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826289.101690:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826289.101697:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346145024:12345-192.168.203.156@tcp:104: Request processed in 102us (285us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826289.101705:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43514 00000100:00100000:3.0:1787826289.139022:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826289.139036:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346146560 00000100:00100000:3.0:1787826289.139041:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346146560 00000100:00100000:3.0:1787826289.139089:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43515 00000100:00100000:3.0:1787826289.139098:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c6d89050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346146560:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826289.139108:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608010b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49c15 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826289.139258:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826289.139266:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c6d89050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346146560:12345-192.168.203.156@tcp:104: Request processed in 169us (1629us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826289.139273:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43515 00000100:00100000:3.0:1787826289.173033:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346147456 00000100:00100000:3.0:1787826289.173044:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346147456 00000100:00100000:3.0:1787826289.173101:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43516 00000100:00100000:3.0:1787826289.173112:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346147456:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826289.173130:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608010ba lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e44:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49c31 expref: -99 pid: 224016 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.173209:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826289.173217:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346147456:12345-192.168.203.156@tcp:104: Request processed in 107us (506us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826289.173261:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43516 00000100:00100000:3.0:1787826289.187652:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346147712 00000100:00100000:3.0:1787826289.187665:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346147712 00000100:00100000:3.0:1787826289.187760:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43517 00000100:00100000:3.0:1787826289.187772:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35094:x1874663346147712:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826289.187790:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e87608010f2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49d0a expref: -99 pid: 224052 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826289.187909:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826289.187925:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35094:x1874663346147712:12345-192.168.203.156@tcp:104: Request processed in 147us (1020us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826289.187932:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43517 00000080:00200000:3.0:1787826289.274543:0:224060:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057c7539000) / proj=0 00000080:00000004:3.0:1787826289.274777:0:224060:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057c7539000: age 8099, max_age 8104 00800000:00000004:3.0:1787826289.274793:0:224060:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: age 8099, max_age 8104 00000100:00100000:3.0:1787826289.274898:0:224060:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:224060:1874663332548864:192.168.203.156@tcp:41:stat.0 00000100:00100000:3.0:1787826289.275187:0:224060:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00010000:00010000:3.0:1787826289.352838:0:224064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.352853:0:224064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826289.352863:0:224064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080111c from it ffffb512c6db3930 00010000:00010000:3.0:1787826289.352867:0:224064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.352875:0:224064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826289.352972:0:224064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826289.352988:0:224064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.352996:0:224064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826289.353004:0:224064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826289.353011:0:224064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826289.353015:0:224064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826289.353019:0:224064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080111c 00000080:00010000:3.0:1787826289.353024:0:224064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080111c from it ffffb512c6db3be8 00010000:00010000:3.0:1787826289.353027:0:224064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826289.353034:0:224064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826289.353042:0:224064:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826289.353048:0:224064:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:3.0:1787826289.353072:0:224064:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826289.353078:0:224064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826289.353085:0:224064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.353091:0:224064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826289.353099:0:224064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826289.353118:0:224064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826289.353122:0:224064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826289.353125:0:224064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080111c 00000080:00010000:3.0:1787826289.353129:0:224064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080111c from it ffffb512c6db3cb8 00010000:00010000:3.0:1787826289.353132:0:224064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826289.353138:0:224064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826289.353711:0:224064:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826289.353717:0:224064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826289.353729:0:224064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.353780:0:224064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e876080112a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.353789:0:224064:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e876080112a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.353798:0:224064:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e876080112a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224064 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.353822:0:224064:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:224064:1874663332549248:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826289.353905:0:224064:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c540 going to sleep for 11 seconds 00000002:00010000:3.0:1787826289.359540:0:224064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.359554:0:224064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826289.359565:0:224064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826289.359572:0:224064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826289.359577:0:224064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826289.359580:0:224064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080111c 00000080:00010000:3.0:1787826289.359585:0:224064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080111c from it ffffb512c6db3bb0 00010000:00010000:3.0:1787826289.359588:0:224064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826289.359595:0:224064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1787826289.359604:0:224064:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 44 00000080:00200008:3.0:1787826289.359612:0:224064:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200080:3.0:1787826289.359631:0:224064:0:(dir.c:2001:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) cmd=c03066f0 arg=7fff83ac6570 00000080:00200000:3.0:1787826289.359652:0:224064:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) name=d45j.sanityn stripe_offset=1 stripe_count=2, hash_type=3 00000020:00100000:3.0:1787826289.359699:0:224064:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826289.359706:0:224064:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb220a80 x1874663332549504/t0(0) o36->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826289.359732:0:224064:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:224064:1874663332549504:192.168.203.156@tcp:36:lfs.0 00000100:00100000:3.0:1787826289.359832:0:224064:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.367427:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826289.367581:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:3.0:1787826289.372302:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826289.372312:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372322:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826289.372326:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372334:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372343:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372472:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372480:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826289.372507:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372516:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080111c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f49df1 expref: -99 pid: 224064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826289.372538:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826289.372540:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826289.372541:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826289.372549:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826289.372562:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332549760:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826289.436668:0:224053:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436679:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436692:0:224053:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436703:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436717:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436726:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436743:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826289.436761:0:224053:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057d0de0000 x1874663332550016/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826305 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826289.436775:0:224053:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436793:0:224053:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7750 (0 0) 00000002:00010000:3.0:1787826289.436809:0:224053:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e46:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826289.436850:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e876080113f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e46:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436857:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e876080113f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.436866:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e876080113f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.436885:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550144:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.436957:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.447706:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550272:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.447742:0:224053:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.447755:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.447766:0:224053:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.447887:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.447905:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.447914:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.448389:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826289.448404:0:224053:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0000 x1874663332550272/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826305 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826289.448420:0:224053:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.448434:0:224053:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/401 (0 0) 00010000:00010000:3.0:1787826289.448443:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.448452:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826289.448484:0:224053:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x18f:0x0]) in obj [0x200000401:0x18f:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.448532:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.448540:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.448550:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.448569:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550400:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.448658:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.457364:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550400:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.457399:0:224053:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 4/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457413:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457424:0:224053:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457434:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457446:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457455:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457468:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826289.457480:0:224053:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222d80 x1874663332550400/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826305 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826289.457495:0:224053:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457508:0:224053:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/399 (0 0) 00010000:00010000:3.0:1787826289.457516:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457525:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826289.457559:0:224053:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e46:0x0], updating i_size 8192 00000080:00200000:3.0:1787826289.457563:0:224053:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e46:0x0](ffff9057dfde3a88): rc = 0 00000080:00010000:3.0:1787826289.457571:0:224053:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde3a88) for remote lock 0x95194e876080113f 00000080:00010000:3.0:1787826289.457575:0:224053:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde3a88) for lock 0x95194e8760801138 00000080:00010000:3.0:1787826289.457586:0:224053:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde3a88) 00000080:00010000:3.0:1787826289.457590:0:224053:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801138 from it ffffb512c6d2bac8 00010000:00010000:3.0:1787826289.457593:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457601:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801138 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e76 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826289.457609:0:224053:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e876080113f from it ffffb512c6d2bac8 00010000:00010000:3.0:1787826289.457613:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e876080113f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e84 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.457622:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e876080113f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e84 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826289.457653:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), inode mode 41ed mask 1 00000080:00200000:3.0:1787826289.457673:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), inode mode 41ed mask 3 00000080:00200000:3.0:1787826289.457679:0:224053:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), file ffff9057c6272000, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1787826289.457686:0:224053:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), intent=open|creat 00000002:00010000:3.0:1787826289.457714:0:224053:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x200000403:0x1db7:0x0]) in obj [0x200000401:0x18f:0x0], intent: open|creat flags 0101102 00010000:00010000:3.0:1787826289.457727:0:224053:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.458048:0:224053:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.458055:0:224053:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826289.458058:0:224053:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e876080114d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f49ea0 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.458093:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x18f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.458099:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826289.458111:0:224053:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826289.458118:0:224053:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb220e00 x1874663332550656/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826289.458127:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.458146:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550656:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.459018:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.467198:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550656:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.467261:0:224053:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/0,1 mode: --/CW res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467292:0:224053:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467301:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467313:0:224053:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467324:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467337:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467350:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467358:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826289.467371:0:224053:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057fb220e00 x1874663332550656/t8589957341(8589957341) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826289.467394:0:224053:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220e00 x1874663332550656/t8589957341(8589957341) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826289.467410:0:224053:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826289.467425:0:224053:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467436:0:224053:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7607 (0 0) 00000080:00200000:3.0:1787826289.467442:0:224053:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000401:0x18f:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826289.467455:0:224053:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), intent=open|creat 00000080:00200000:3.0:1787826289.467484:0:224053:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1db7:0x0](ffff9057dfde9988) 00000080:00200000:3.0:1787826289.467560:0:224053:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1db7:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826289.467566:0:224053:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000403:0x1db7:0x0](ffff9057dfde9988) 00000080:00010000:3.0:1787826289.467569:0:224053:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1db7:0x0](ffff9057dfde9988) for lock 0x95194e8760801154 00000080:00010000:3.0:1787826289.467577:0:224053:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1db7:0x0](ffff9057dfde9988) 00000080:00200008:3.0:1787826289.467589:0:224053:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1db7:0x0](ffff9057dfde9988), flags 101101 00000002:00100000:3.0:1787826289.467609:0:224053:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220e00 x1874663332550656/t8589957341(8589957341) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826289.467656:0:224053:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801154 from it ffff9057edecc000 00010000:00010000:3.0:1787826289.467661:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467671:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801154 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1db7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f49ebc expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826289.467693:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1db7:0x0](ffff9057dfde9988), inode mode 81a4 mask 40 00000080:00200008:3.0:1787826289.467770:0:224053:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn:[0x200000403:0x1db7:0x0], ppos: 0, count: 6 00000080:00200000:3.0:1787826289.467781:0:224053:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:3.0:1787826289.467787:0:224053:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:3.0:1787826289.467792:0:224053:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: write ppos: 0, bytes: 6 00000080:00200000:3.0:1787826289.467796:0:224053:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826289.467804:0:224053:0:(cl_io.c:214:cl_io_rw_init()) header@0000000088daa763[0x110, 1, [0x200000403:0x1db7:0x0] hash] 00000020:00200000:3.0:1787826289.467808:0:224053:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:3.0:1787826289.467814:0:224053:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db7:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826289.467828:0:224053:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1787826289.467837:0:224053:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:3.0:1787826289.467841:0:224053:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826289.467860:0:224053:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1787826289.467875:0:224053:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826289.467880:0:224053:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a715b0, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826289.467927:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467936:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.467946:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.467965:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550784:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.468040:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.472804:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332550784:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.472831:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.472849:0:224053:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.472860:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.472874:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826289.472887:0:224053:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.472968:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.472978:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826289.473083:0:224053:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn: write [0, 6) append 0 append lockpos 0 00000080:00200000:3.0:1787826289.473103:0:224053:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:3.0:1787826289.473152:0:224053:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:3.0:1787826289.473162:0:224053:0:(rw26.c:929:ll_write_end()) page@000000009f22829f[3 000000002df757c4 2 1 000000000bf0707e 0 0 0] 00000080:00200000:3.0:1787826289.473166:0:224053:0:(rw26.c:929:ll_write_end()) vmpage @00000000bc5ab157 17ffffc0001001 3:0 ffff9057f9906540 0 lru 00000080:00200000:3.0:1787826289.473175:0:224053:0:(rw26.c:929:ll_write_end()) osc-page@00000000a1bcbb5f 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009958d8f2 0000000030dfc484 > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1787826289.473177:0:224053:0:(rw26.c:929:ll_write_end()) end page@000000009f22829f 00000080:00200000:3.0:1787826289.473178:0:224053:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1787826289.473182:0:224053:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:3.0:1787826289.473201:0:224053:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826289.473314:0:224053:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfde9b38, count 1, dirtied 1 00000080:00200000:3.0:1787826289.473321:0:224053:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:3.0:1787826289.473324:0:224053:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn: write bytes 0, result: 6 00000020:00010000:3.0:1787826289.473342:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_release():245 00000020:00010000:3.0:1787826289.473346:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826289.473349:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a715b0 (ffff9057f989c518/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826289.473355:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.473366:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826289.473382:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826289.473385:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a715b0 (ffff9057d889c878/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826289.473392:0:224053:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826289.473400:0:224053:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1787826289.473410:0:224053:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db7:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826289.473414:0:224053:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826289.473422:0:224053:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1db7:0x0] updating i_size 6 i_blocks 0 00000080:00200000:3.0:1787826289.473427:0:224053:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:3.0:1787826289.473461:0:224053:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1db7:0x0](ffff9057dfde9988), flags 100001 00000002:00100000:3.0:1787826289.473670:0:224053:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb220e00 x1874663332550656/t8589957341(8589957341) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826289.473682:0:224053:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826289.473687:0:224053:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222d80 x1874663332551040/t0(0) o35->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826289.473707:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551040:192.168.203.156@tcp:35:bash.0 00000100:00100000:3.0:1787826289.474327:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.482311:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551040:192.168.203.156@tcp:35:bash.0 00000002:00100000:3.0:1787826289.482351:0:224053:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb220e00 x1874663332550656/t8589957341(8589957341) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826289.482563:0:224053:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826289.482581:0:224053:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826289.482599:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07c00/0x95194e8760801131 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e5a expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.482613:0:224053:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07c00/0x95194e8760801131 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e5a expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826289.482622:0:224053:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826289.482627:0:224053:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826289.482631:0:224053:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801131 00000080:00010000:3.0:1787826289.482638:0:224053:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801131 from it ffffb512c6d2bbf8 00010000:00010000:3.0:1787826289.482651:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07c00/0x95194e8760801131 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49e5a expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.482661:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07c00/0x95194e8760801131 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49e5a expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826289.482676:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826289.482683:0:224053:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=25 00000080:00200000:3.0:1787826289.482690:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), inode mode 41ed mask 1 00000080:00200000:3.0:1787826289.482702:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), inode mode 41ed mask 3 00000080:00200000:3.0:1787826289.482708:0:224053:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), file ffff9057d1775c80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1787826289.482717:0:224053:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), intent=open|creat 00000002:00010000:3.0:1787826289.482739:0:224053:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x240000403:0x1e47:0x0]) in obj [0x240000400:0x191:0x0], intent: open|creat flags 0101102 00010000:00010000:3.0:1787826289.482753:0:224053:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.482802:0:224053:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.482809:0:224053:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826289.482812:0:224053:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b06800/0x95194e8760801146 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f49e92 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826289.482846:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x191:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.482854:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826289.482866:0:224053:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826289.482874:0:224053:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb221f80 x1874663332551296/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826289.482882:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224053 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.482912:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551296:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.483068:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef30c3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.493958:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551296:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.493984:0:224053:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/0,1 mode: --/CW res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494027:0:224053:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494042:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494055:0:224053:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494066:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494080:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494094:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494111:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826289.494123:0:224053:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057fb221f80 x1874663332551296/t4294985357(4294985357) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826289.494136:0:224053:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb221f80 x1874663332551296/t4294985357(4294985357) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826289.494150:0:224053:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826289.494163:0:224053:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494330:0:224053:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7751 (0 0) 00000080:00200000:3.0:1787826289.494346:0:224053:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000400:0x191:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826289.494360:0:224053:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e46:0x0](ffff9057dfde3a88), intent=open|creat 00000080:00200000:3.0:1787826289.494389:0:224053:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e47:0x0](ffff9057dfdeb248) 00000080:00200000:3.0:1787826289.494466:0:224053:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e47:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826289.494471:0:224053:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240000403:0x1e47:0x0](ffff9057dfdeb248) 00000080:00010000:3.0:1787826289.494474:0:224053:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e47:0x0](ffff9057dfdeb248) for lock 0x95194e8760801162 00000080:00010000:3.0:1787826289.494484:0:224053:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e47:0x0](ffff9057dfdeb248) 00000080:00200008:3.0:1787826289.494497:0:224053:0:(file.c:1090:ll_file_open()) START file f45j.sanityn-2:[0x240000403:0x1e47:0x0](ffff9057dfdeb248), flags 101101 00000002:00100000:3.0:1787826289.494519:0:224053:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb221f80 x1874663332551296/t4294985357(4294985357) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826289.494556:0:224053:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801162 from it ffff9057c83f8ea0 00010000:00010000:3.0:1787826289.494562:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494571:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801162 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f49ee6 expref: -99 pid: 224053 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826289.494593:0:224053:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e47:0x0](ffff9057dfdeb248), inode mode 81a4 mask 40 00000080:00200008:3.0:1787826289.494660:0:224053:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn-2:[0x240000403:0x1e47:0x0], ppos: 0, count: 6 00000080:00200000:3.0:1787826289.494671:0:224053:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:3.0:1787826289.494677:0:224053:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:3.0:1787826289.494682:0:224053:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn-2: write ppos: 0, bytes: 6 00000080:00200000:3.0:1787826289.494687:0:224053:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn-2: desiginated mirror: 0 00000020:00200000:3.0:1787826289.494694:0:224053:0:(cl_io.c:214:cl_io_rw_init()) header@00000000e126f29f[0x110, 1, [0x240000403:0x1e47:0x0] hash] 00000020:00200000:3.0:1787826289.494696:0:224053:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:3.0:1787826289.494701:0:224053:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e47:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826289.494715:0:224053:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:3.0:1787826289.494723:0:224053:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:3.0:1787826289.494727:0:224053:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826289.494744:0:224053:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:3.0:1787826289.494758:0:224053:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826289.494762:0:224053:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a715b0, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826289.494804:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494815:0:224053:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.494826:0:224053:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826289.494845:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551424:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826289.494922:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c8b87540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.504613:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551424:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826289.504650:0:224053:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504668:0:224053:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504680:0:224053:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504694:0:224053:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826289.504706:0:224053:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504718:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504726:0:224053:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826289.504759:0:224053:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn-2: write [0, 6) append 0 append lockpos 0 00000080:00200000:3.0:1787826289.504777:0:224053:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:3.0:1787826289.504829:0:224053:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:3.0:1787826289.504840:0:224053:0:(rw26.c:929:ll_write_end()) page@000000007428ba67[3 00000000169cc264 2 1 000000000bf0707e 0 0 0] 00000080:00200000:3.0:1787826289.504843:0:224053:0:(rw26.c:929:ll_write_end()) vmpage @000000002361c4d9 17ffffc0001001 3:0 ffff9057f99069a0 0 lru 00000080:00200000:3.0:1787826289.504851:0:224053:0:(rw26.c:929:ll_write_end()) osc-page@0000000097213fcc 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 0000000072dd0513 000000006918ec9d > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1787826289.504853:0:224053:0:(rw26.c:929:ll_write_end()) end page@000000007428ba67 00000080:00200000:3.0:1787826289.504855:0:224053:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:3.0:1787826289.504858:0:224053:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:3.0:1787826289.504883:0:224053:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000001d56ca22 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826289.504901:0:224053:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfdeb3f8, count 1, dirtied 1 00000080:00200000:3.0:1787826289.504905:0:224053:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:3.0:1787826289.504908:0:224053:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn-2: write bytes 0, result: 6 00000020:00010000:3.0:1787826289.504923:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_release():245 00000020:00010000:3.0:1787826289.504927:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826289.504931:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a715b0 (ffff9057f989c518/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826289.504937:0:224053:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826289.504947:0:224053:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826289.504963:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2f38 (ffff9057d889c878/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826289.504966:0:224053:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a715b0 (ffff9057d889c878/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826289.504974:0:224053:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826289.504981:0:224053:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:3.0:1787826289.504989:0:224053:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e47:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826289.504992:0:224053:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn-2: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826289.505000:0:224053:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e47:0x0] updating i_size 6 i_blocks 0 00000080:00200000:3.0:1787826289.505005:0:224053:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:3.0:1787826289.505050:0:224053:0:(file.c:479:ll_file_release()) START file f45j.sanityn-2:[0x240000403:0x1e47:0x0](ffff9057dfdeb248), flags 100001 00000002:00100000:3.0:1787826289.505125:0:224053:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb221f80 x1874663332551296/t4294985357(4294985357) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826289.505136:0:224053:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826289.505142:0:224053:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb223800 x1874663332551680/t0(0) o35->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826289.505162:0:224053:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551680:192.168.203.156@tcp:35:bash.0 00000100:00100000:3.0:1787826289.505265:0:224053:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c8b87540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826289.512327:0:224053:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224053:1874663332551680:192.168.203.156@tcp:35:bash.0 00000002:00100000:3.0:1787826289.512366:0:224053:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb221f80 x1874663332551296/t4294985357(4294985357) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826305 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826289.531157:0:224067:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00000100:00100000:3.0:1787826289.538469:0:6634:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:398e0287-d3c7-44dd-b1d9-6b85f698e615:6634:1874663332551936:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826289.538495:0:6634:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608010e4 lrc: 0/0,0 mode: --/CR res: [0x200000403:0x1db6:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f49cd2 expref: -99 pid: 224014 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826289.538535:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826289.538537:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826289.538539:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826289.571129:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826289.571147:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826289.571176:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826292.942709:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346153472 00000100:00100000:3.0:1787826292.942724:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346153472 00000100:00100000:3.0:1787826292.942812:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43520 00000100:00100000:3.0:1787826292.942823:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346153472:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826292.942847:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000067fcfabb/0x95194e8760801082 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f49b97 expref: -99 pid: 223833 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826292.943580:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826292.943600:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346153472:12345-192.168.203.156@tcp:104: Request processed in 773us (1683us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826292.943612:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43520 00000100:00100000:3.0:1787826293.160074:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332552960:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.166016:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332553088:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.166038:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973839 gen 19 00000100:00100000:3.0:1787826293.166059:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332553600:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.166072:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332553216:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.166086:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332553728:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.168257:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332552832:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.168283:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057c7539000: committing for last_committed 4294985358 gen 1 00000100:00100000:3.0:1787826293.168315:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332553472:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826293.177769:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332553344:192.168.203.156@tcp:400:kworker.0 00010000:00010000:3.0:1787826293.540476:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826293.660161:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332554880:192.168.203.156@tcp:103:mrename.0 00000080:00200000:3.0:1787826293.675276:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826293.675302:0:136971:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826293.675314:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826293.675348:0:136971:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001d56ca22 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826293.675369:0:136971:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001d56ca22 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826293.675437:0:136971:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826293.675526:0:136971:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb223800 - object 0x280000400:1227 offset 0<>6 00000100:00100000:3.0:1787826293.678778:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332555648:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826293.680485:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826293.680530:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000dddf5e42 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826293.680568:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332555264:192.168.203.156@tcp:4:bash.0 00000008:00010000:3.0:1787826293.695315:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001d56ca22 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826293.695369:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001d56ca22 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000ca879f03/0x95194e8760801169 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49ef4 expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826293.695606:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332555520:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826293.695669:0:6159:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826293.695690:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346155136 00000100:00100000:3.0:1787826293.695699:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346155136 00000100:00100000:3.0:1787826293.695762:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43522 00000100:00100000:3.0:1787826293.695773:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c6d8a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8444:x1874663346155136:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826293.695888:0:6159:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826293.695900:0:6159:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826293.695930:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@0000000088daa763[0x110, 2, [0x200000403:0x1db7:0x0] hash] 00000020:00010000:3.0:1787826293.695933:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826289 atime: 1787826289 ctime: 1787826289 blocks: 8 00000020:00010000:3.0:1787826293.695937:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@000000000f77a818[0x110, 2, [0x2c0000401:0x51e:0x0] hash] 00000020:00010000:3.0:1787826293.695939:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826289 atime: 1787826289 ctime: 1787826289 blocks: 8 00000100:00100000:3.0:1787826293.696354:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c6d8a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8444:x1874663346155136:12345-192.168.203.156@tcp:106: Request processed in 578us (739us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826293.696362:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43522 00000100:00100000:3.0:1787826293.717672:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346155392 00000100:00100000:3.0:1787826293.717689:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346155392 00000100:00100000:3.0:1787826293.717760:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43523 00000100:00100000:3.0:1787826293.717771:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346155392:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826293.717788:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d800/0x95194e876080118c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49f4f expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826293.717874:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826293.717881:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346155392:12345-192.168.203.156@tcp:104: Request processed in 112us (357us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826293.717943:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43523 00000100:00100000:3.0:1787826293.718008:0:2395:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973841 gen 19 00000100:00100000:3.0:1787826293.718049:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332555648:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826293.719966:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826293.719995:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346155520 00000100:00100000:3.0:1787826293.720003:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346155520 00000100:00100000:3.0:1787826293.720043:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43524 00000100:00100000:3.0:1787826293.720053:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8759:x1874663346155520:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826293.720067:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b3256fde/0x95194e876080115b lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f49eca expref: -99 pid: 224053 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826293.720140:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826293.720147:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8759:x1874663346155520:12345-192.168.203.156@tcp:104: Request processed in 95us (186us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826293.720154:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43524 00000100:00100000:3.0:1787826293.730676:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826293.747350:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826294.595923:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:3.0:1787826294.957214:0:224272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826294.957273:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.957293:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.957310:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826294.957318:0:224272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826294.957325:0:224272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826294.957329:0:224272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801170 00000080:00010000:3.0:1787826294.957337:0:224272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801170 from it ffffb512c6fabb58 00010000:00010000:3.0:1787826294.957341:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.957350:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826294.957367:0:224272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826294.957374:0:224272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:3.0:1787826294.957383:0:224272:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e46:0x0](ffff9057dfde1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826294.957388:0:224272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde1988),name=d45j.sanityn 00000002:00010000:3.0:1787826294.957400:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e46:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.957408:0:224272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7750 (0 0) 00010000:00010000:3.0:1787826294.957456:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.957464:0:224272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.957477:0:224272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826294.957502:0:224272:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557056:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826294.957617:0:224272:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ede45480 going to sleep for 11 seconds 00000100:00100000:3.0:1787826294.962709:0:224272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557056:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826294.962738:0:224272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962765:0:224272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962774:0:224272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962785:0:224272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962796:0:224272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962823:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962833:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962847:0:224272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826294.962860:0:224272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220a80 x1874663332557056/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826310 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826294.962876:0:224272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962887:0:224272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7750 (0 0) 00000080:00200000:3.0:1787826294.962893:0:224272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826294.962924:0:224272:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e46:0x0], updating i_size 4096 00000080:00010000:3.0:1787826294.962932:0:224272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde1988) for lock 0x95194e87608011cb 00000080:00010000:3.0:1787826294.962937:0:224272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde1988) 00000080:00010000:3.0:1787826294.962940:0:224272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608011cb from it ffffb512c6fabcb8 00010000:00010000:3.0:1787826294.962943:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.962953:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826294.962985:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x191:0x0]) in obj [0x240000400:0x191:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.963028:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x191:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.963036:0:224272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.963048:0:224272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826294.963065:0:224272:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557184:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826294.963134:0:224272:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ede45480 going to sleep for 11 seconds 00000100:00100000:3.0:1787826294.967794:0:224272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557184:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826294.967833:0:224272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x191:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967849:0:224272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967864:0:224272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967886:0:224272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967900:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967909:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967918:0:224272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826294.967937:0:224272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220a80 x1874663332557184/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826310 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826294.967963:0:224272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.967984:0:224272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/401 (0 0) 00010000:00010000:3.0:1787826294.968001:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.968010:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826294.968048:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x18f:0x0]) in obj [0x200000401:0x18f:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.969687:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.969700:0:224272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.969710:0:224272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826294.969744:0:224272:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557312:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826294.969867:0:224272:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ede45480 going to sleep for 11 seconds 00000100:00100000:3.0:1787826294.982390:0:224272:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057d128a800: committing for last_committed 8589957348 gen 1 00000100:00100000:3.0:1787826294.982410:0:224272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557312:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826294.982436:0:224272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x18f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982449:0:224272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982461:0:224272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982473:0:224272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982558:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982568:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982577:0:224272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826294.982589:0:224272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220a80 x1874663332557312/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826310 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826294.982615:0:224272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982630:0:224272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/399 (0 0) 00010000:00010000:3.0:1787826294.982639:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.982650:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826294.982738:0:224272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826294.982759:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.982770:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.982906:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826294.982916:0:224272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826294.982923:0:224272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826294.982927:0:224272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801170 00000080:00010000:3.0:1787826294.982931:0:224272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801170 from it ffffb512c6fabbf8 00010000:00010000:3.0:1787826294.982935:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.982942:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826294.982950:0:224272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826294.982956:0:224272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:3.0:1787826294.982963:0:224272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde1988), inode mode 41ed mask 44 00000080:00200008:3.0:1787826294.982972:0:224272:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e46:0x0](ffff9057dfde1988), flags 704400 00000002:00010000:3.0:1787826294.982988:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e46:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1787826294.983177:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.983185:0:224272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826294.983197:0:224272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826294.983204:0:224272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb220a80 x1874663332557568/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826294.983214:0:224272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826294.983260:0:224272:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557568:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826294.983512:0:224272:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ede45480 going to sleep for 11 seconds 00000100:00100000:3.0:1787826294.989600:0:224272:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224272:1874663332557568:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826294.989633:0:224272:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989648:0:224272:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989655:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989663:0:224272:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989672:0:224272:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989684:0:224272:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826294.989689:0:224272:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989707:0:224272:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989714:0:224272:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989721:0:224272:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826294.989727:0:224272:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826294.989734:0:224272:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e87608011e0 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224272 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826294.989762:0:224272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb220a80 x1874663332557568/t4294985361(4294985361) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826310 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826294.989776:0:224272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220a80 x1874663332557568/t4294985361(4294985361) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826310 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826294.989997:0:224272:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826294.990030:0:224272:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e46:0x0], updating i_size 4096 00000002:00100000:3.0:1787826294.990050:0:224272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220a80 x1874663332557568/t4294985361(4294985361) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826310 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826294.990141:0:224272:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e46:0x0](ffff9057dfde1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826294.990149:0:224272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde1988),name=d45j.sanityn 00000002:00010000:3.0:1787826294.990167:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e46:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.990180:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990193:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826294.990203:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x191:0x0]) in obj [0x240000400:0x191:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.990210:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990218:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990275:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990284:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826294.990298:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x18f:0x0]) in obj [0x200000401:0x18f:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.990305:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990314:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990322:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990330:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826294.990340:0:224272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826294.990345:0:224272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde1988) 00000080:00010000:3.0:1787826294.990349:0:224272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e46:0x0](ffff9057dfde1988) for lock 0x95194e87608011cb 00000080:00010000:3.0:1787826294.990353:0:224272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608011cb from it ffffb512c6fabcd8 00010000:00010000:3.0:1787826294.990356:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826294.990363:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e87608011cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a052 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826294.990375:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x191:0x0]) in obj [0x240000400:0x191:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.990381:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990388:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990396:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990403:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:3.0:1787826294.990410:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x18f:0x0]) in obj [0x200000401:0x18f:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826294.990415:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990423:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990430:0:224272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826294.990438:0:224272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826294.990507:0:224272:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e46:0x0](ffff9057dfde1988) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826294.990521:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:3.0:1787826294.990545:0:224272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x191:0x0]) in obj [0x240000400:0x191:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826294.990554:0:224272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826294.990562:0:224272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787826295.075671:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346161664 00000100:00100000:3.0:1787826295.075680:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346161664 00000100:00100000:3.0:1787826295.075734:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43528 00000100:00100000:3.0:1787826295.075745:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346161664:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.075763:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.075836:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.075844:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346161664:12345-192.168.203.156@tcp:104: Request processed in 101us (260us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.075851:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43528 00010000:00010000:3.0:1787826295.075884:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826295.075890:0:15673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.075906:0:15673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826295.075911:0:15673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.075920:0:15673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.075929:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.076053:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.076307:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826295.076327:0:15673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.076335:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608011d9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x18f:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4a06e expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.076356:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826295.076358:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826295.076360:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826295.076375:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332559232:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826295.084548:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332559232:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826295.109691:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826295.109710:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346163200 00000100:00100000:3.0:1787826295.109720:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346163200 00000100:00100000:3.0:1787826295.109775:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43529 00000100:00100000:3.0:1787826295.109787:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d9e6b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346163200:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.109804:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e876080119a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49f72 expref: -99 pid: 224236 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826295.109877:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.109885:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d9e6b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346163200:12345-192.168.203.156@tcp:104: Request processed in 100us (318us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.109893:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43529 00000100:00100000:3.0:1787826295.113583:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346163328 00000100:00100000:3.0:1787826295.113593:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346163328 00000100:00100000:3.0:1787826295.113656:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43530 00000100:00100000:3.0:1787826295.113666:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346163328:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.113683:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e8760801170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49f09 expref: -99 pid: 224234 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826295.113763:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.113770:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346163328:12345-192.168.203.156@tcp:104: Request processed in 106us (285us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.113778:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43530 00000100:00100000:3.0:1787826295.138726:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346164096 00000100:00100000:3.0:1787826295.138737:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346164096 00000100:00100000:3.0:1787826295.138795:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43531 00000100:00100000:3.0:1787826295.138806:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346164096:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.138824:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07c00/0x95194e87608011a1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f49f8e expref: -99 pid: 224236 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826295.138975:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.138984:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346164096:12345-192.168.203.156@tcp:104: Request processed in 180us (478us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.138991:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43531 00000100:00100000:3.0:1787826295.140712:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332560256:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826295.163853:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332560256:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826295.166531:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346164352 00000100:00100000:3.0:1787826295.166551:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346164352 00000100:00100000:3.0:1787826295.166691:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43532 00000100:00100000:3.0:1787826295.166705:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35098:x1874663346164352:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.166733:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e87608011d2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a060 expref: -99 pid: 224272 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826295.166858:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.166865:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35098:x1874663346164352:12345-192.168.203.156@tcp:104: Request processed in 164us (1444us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.166880:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43532 00000080:00200000:3.0:1787826295.303529:0:224280:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057c7539000) / proj=0 00000080:00000004:3.0:1787826295.303539:0:224280:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057c7539000: age 8105, max_age 8110 00800000:00000004:3.0:1787826295.303553:0:224280:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: age 8105, max_age 8110 00000100:00100000:3.0:1787826295.303611:0:224280:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:224280:1874663332560512:192.168.203.156@tcp:41:stat.0 00000100:00100000:3.0:1787826295.303820:0:224280:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c78743c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826295.311287:0:224280:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957349 gen 1 00000100:00100000:3.0:1787826295.311326:0:224280:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:224280:1874663332560512:192.168.203.156@tcp:41:stat.0 00800000:00000004:3.0:1787826295.311354:0:224280:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: update ffff9057c6ef0450 cache blocks 1793351/1916558 objects 519494/534189 00000080:00000004:3.0:1787826295.311363:0:224280:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057c7539000: update ffff9057c6ef37c0 cache blocks 1793351/1916558 objects 519494/534189 00000080:00000004:3.0:1787826295.311368:0:224280:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793351/1916558 objects 519494/534189 00000100:00100000:3.0:1787826295.420423:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346166144 00000100:00100000:3.0:1787826295.420428:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346166144 00000100:00100000:3.0:1787826295.420465:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43533 00000100:00100000:3.0:1787826295.420475:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346166144:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826295.420504:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e87608011fc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a147 expref: -99 pid: 224284 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826295.420575:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.420583:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346166144:12345-192.168.203.156@tcp:104: Request processed in 109us (531us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826295.420595:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43533 00010000:00010000:3.0:1787826295.609273:0:224289:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826295.609536:0:224289:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826295.614273:0:224289:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1787826295.629593:0:6166:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:6166:1874663332563584:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826295.629643:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2800/0x95194e8760801242 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4a23c expref: -99 pid: 224273 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826295.629676:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e876080121f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e48:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4a1da expref: -99 pid: 224273 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826295.629691:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b07400/0x95194e87608011a8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e46:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f49f9c expref: -99 pid: 224236 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826295.629719:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826295.629721:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826295.629724:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826298.814333:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332565248:192.168.203.156@tcp:16:kworker.0 00000080:00200000:3.0:1787826298.814398:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1787826298.823558:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006cff921b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801249 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a24a expref: -99 pid: 224273 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826298.823622:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006cff921b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801249 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4c9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a24a expref: -99 pid: 224273 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826298.823666:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332565120:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826299.082803:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826300.235703:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826300.240720:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826300.253754:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826300.260608:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826300.263278:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346173824 00000100:00100000:3.0:1787826300.263293:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346173824 00000100:00100000:3.0:1787826300.263372:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43539 00000100:00100000:3.0:1787826300.263384:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb223800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8759:x1874663346173824:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826300.263408:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000a743de22/0x95194e876080123b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4a220 expref: -99 pid: 224273 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826300.263741:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826300.263751:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb223800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8759:x1874663346173824:12345-192.168.203.156@tcp:104: Request processed in 368us (3158us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826300.263759:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43539 00000100:00100000:3.0:1787826300.267287:0:224455:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224455:1874663332567680:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826300.267332:0:224455:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826300.267352:0:224455:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826300.267369:0:224455:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826300.277333:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000020:00010000:3.0:1787826300.288580:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826300.288585:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826300.288589:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9bb25b0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826300.288603:0:224455:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826300.288621:0:224455:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826300.288642:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826300.288645:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9bb25b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826300.288654:0:224455:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826300.288668:0:224455:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826300.288673:0:224455:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826300.288679:0:224455:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:3.0:1787826300.288706:0:224455:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1db8:0x0], ppos: 6, count: 4194304 00000080:00200000:3.0:1787826300.288717:0:224455:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:3.0:1787826300.288723:0:224455:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826300.288730:0:224455:0:(cl_io.c:214:cl_io_rw_init()) header@00000000e728ee0d[0x110, 2, [0x200000403:0x1db8:0x0] hash] 00000020:00200000:3.0:1787826300.288733:0:224455:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:3.0:1787826300.288737:0:224455:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826300.288760:0:224455:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:3.0:1787826300.288765:0:224455:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826300.288784:0:224455:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:3.0:1787826300.288799:0:224455:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826300.288805:0:224455:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71c30, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826300.288831:0:224455:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288847:0:224455:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288859:0:224455:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288869:0:224455:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288876:0:224455:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826300.288885:0:224455:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:3.0:1787826300.288902:0:224455:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1db8:0x0] 00000008:00010000:3.0:1787826300.288907:0:224455:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826300.288909:0:224455:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a715b0, osc lock ffff9057c50ba850, flags 0x400001002 00010000:00010000:3.0:1787826300.288916:0:224455:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288926:0:224455:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826300.288938:0:224455:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826300.288941:0:224455:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.288947:0:224455:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826300.288953:0:224455:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826300.288958:0:224455:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1db8:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826300.288961:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826300.288963:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826300.288965:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a715b0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826300.288968:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826300.288970:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a715b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826300.288989:0:224455:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:3.0:1787826300.288994:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826300.288997:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826300.288999:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71c30 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826300.289003:0:224455:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826300.289010:0:224455:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826300.289018:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826300.289020:0:224455:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71c30 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826300.289024:0:224455:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1787826300.289029:0:224455:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826300.289031:0:224455:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1787826300.289034:0:224455:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1787826300.289149:0:224455:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1db8:0x0](ffff9057dfdfcb08), flags 100000 00000002:00100000:3.0:1787826300.289287:0:224455:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb239f80 x1874663332566272/t8589957360(8589957360) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826315 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826300.289299:0:224455:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826300.289306:0:224455:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb220000 x1874663332568320/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826300.289331:0:224455:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224455:1874663332568320:192.168.203.156@tcp:35:cat.0 00000100:00100000:3.0:1787826300.289443:0:224455:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826300.291205:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346174720 00000100:00100000:3.0:1787826300.291216:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346174720 00000100:00100000:3.0:1787826300.291316:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43541 00000100:00100000:3.0:1787826300.291328:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125380:x1874663346174720:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826300.291349:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2ce00/0x95194e8760801273 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1db8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4a338 expref: -99 pid: 224455 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826300.291432:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826300.291439:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125380:x1874663346174720:12345-192.168.203.156@tcp:104: Request processed in 113us (426us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826300.291447:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43541 00000100:00100000:3.0:1787826300.671942:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346177152 00000100:00100000:3.0:1787826300.671958:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346177152 00000100:00100000:3.0:1787826300.672097:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43542 00000100:00100000:3.0:1787826300.672109:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346177152:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826300.672132:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002bb75d1f/0x95194e87608012a4 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x51e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4a369 expref: -99 pid: 224455 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826300.672244:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826300.672252:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346177152:12345-192.168.203.156@tcp:104: Request processed in 145us (440us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826300.672259:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43542 00010000:00010000:3.0:1787826300.713045:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826301.657763:0:224493:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826301.657808:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.658085:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.658105:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826301.658115:0:224493:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826301.658125:0:224493:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826301.658130:0:224493:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801250 00000080:00010000:3.0:1787826301.658139:0:224493:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801250 from it ffffb512c713bb58 00010000:00010000:3.0:1787826301.658142:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.658153:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826301.658172:0:224493:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826301.658182:0:224493:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:3.0:1787826301.658191:0:224493:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e48:0x0](ffff9057dfdd2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826301.658196:0:224493:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e48:0x0](ffff9057dfdd2a08),name=d45j.sanityn 00000002:00010000:3.0:1787826301.658213:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e48:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.658661:0:224493:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7752 (0 0) 00010000:00010000:3.0:1787826301.658734:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.658742:0:224493:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.658754:0:224493:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826301.658785:0:224493:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224493:1874663332568704:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826301.659072:0:224493:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:3.0:1787826301.698135:0:224493:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239f80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224493:1874663332569216:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826301.698166:0:224493:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698181:0:224493:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698189:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698196:0:224493:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698207:0:224493:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698214:0:224493:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826301.698252:0:224493:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698261:0:224493:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698269:0:224493:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698277:0:224493:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826301.698285:0:224493:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826301.698292:0:224493:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e87608012c0 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224493 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826301.698321:0:224493:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb239f80 x1874663332569216/t4294985372(4294985372) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826317 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826301.698335:0:224493:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb239f80 x1874663332569216/t4294985372(4294985372) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826317 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826301.698353:0:224493:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826301.698393:0:224493:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e48:0x0], updating i_size 4096 00000002:00100000:3.0:1787826301.698417:0:224493:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb239f80 x1874663332569216/t4294985372(4294985372) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826317 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826301.698512:0:224493:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e48:0x0](ffff9057dfdd2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826301.698519:0:224493:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e48:0x0](ffff9057dfdd2a08),name=d45j.sanityn 00000002:00010000:3.0:1787826301.698544:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e48:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.698569:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e48:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3a8 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698582:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e48:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3a8 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826301.698594:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x192:0x0]) in obj [0x240000400:0x192:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.698601:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698608:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698617:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698624:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826301.698639:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x190:0x0]) in obj [0x200000401:0x190:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.698647:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698655:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698663:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698670:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826301.698680:0:224493:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826301.698686:0:224493:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e48:0x0](ffff9057dfdd2a08) 00000080:00010000:3.0:1787826301.698691:0:224493:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e48:0x0](ffff9057dfdd2a08) for lock 0x95194e87608012ab 00000080:00010000:3.0:1787826301.698696:0:224493:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608012ab from it ffffb512c713bcd8 00010000:00010000:3.0:1787826301.698700:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e48:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3a8 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826301.698707:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e87608012ab lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e48:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3a8 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826301.698716:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x192:0x0]) in obj [0x240000400:0x192:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.698721:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698729:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698736:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698742:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:3.0:1787826301.698748:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x190:0x0]) in obj [0x200000401:0x190:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826301.698753:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698760:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698767:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.698772:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826301.698865:0:224493:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e48:0x0](ffff9057dfdd2a08) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826301.698880:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:3.0:1787826301.698906:0:224493:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x192:0x0]) in obj [0x240000400:0x192:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826301.698913:0:224493:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826301.698921:0:224493:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787826301.699709:0:224493:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224493:1874663332569344:192.168.203.156@tcp:37:rm.0 00000100:00100000:3.0:1787826301.699853:0:224493:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51d80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826301.711797:0:224493:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224493:1874663332569344:192.168.203.156@tcp:37:rm.0 00010000:00010000:3.0:1787826301.712874:0:224493:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826301.712897:0:224493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608012b2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a3b6 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826301.790344:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826301.790371:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346177664 00000100:00100000:3.0:1787826301.790385:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346177664 00000100:00100000:3.0:1787826301.790459:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43543 00000100:00100000:3.0:1787826301.790472:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d86dc450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346177664:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826301.790503:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e87608012b9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x190:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a3c4 expref: -99 pid: 224493 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826301.790604:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826301.790612:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d86dc450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346177664:12345-192.168.203.156@tcp:104: Request processed in 143us (654us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826301.790620:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43543 00000100:00100000:3.0:1787826301.834519:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346179200 00000100:00100000:3.0:1787826301.834534:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346179200 00000100:00100000:3.0:1787826301.834611:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346179328 00000100:00100000:3.0:1787826301.834615:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346179328 00000100:00100000:3.0:1787826301.834636:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43544 00000100:00100000:3.0:1787826301.834648:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346179200:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826301.834672:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07400/0x95194e876080127a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a2c8 expref: -99 pid: 224457 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826301.834768:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826301.834776:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346179200:12345-192.168.203.156@tcp:104: Request processed in 130us (422us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826301.834786:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43544 00000100:00100000:3.0:1787826301.834807:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43545 00000100:00100000:3.0:1787826301.834814:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346179328:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826301.834823:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a25f expref: -99 pid: 224455 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826301.834875:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826301.834882:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346179328:12345-192.168.203.156@tcp:104: Request processed in 69us (459us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826301.834888:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43545 00000100:00100000:3.0:1787826301.847635:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826301.862181:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826302.101879:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346182144 00000100:00100000:3.0:1787826302.101884:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346182144 00000100:00100000:3.0:1787826302.101932:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43548 00000100:00100000:3.0:1787826302.101942:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346182016:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826302.101960:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608012ea lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a4c0 expref: -99 pid: 224505 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826302.102078:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826302.102087:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346182016:12345-192.168.203.156@tcp:104: Request processed in 146us (302us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826302.102103:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43548 00000100:00100000:3.0:1787826302.182638:0:224494:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224494:1874663332573184:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826302.182667:0:224494:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182692:0:224494:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182703:0:224494:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182715:0:224494:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182728:0:224494:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182737:0:224494:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182745:0:224494:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826302.182757:0:224494:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de1500 x1874663332573184/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826318 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826302.182771:0:224494:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182785:0:224494:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826302.182791:0:224494:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826302.182818:0:224494:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826302.182824:0:224494:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608012f1 00000080:00010000:3.0:1787826302.182830:0:224494:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826302.182833:0:224494:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608012f1 from it ffffb512c716bbf8 00010000:00010000:3.0:1787826302.182837:0:224494:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182846:0:224494:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06200/0x95194e87608012f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a506 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826302.182872:0:224494:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826302.182893:0:224494:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:3.0:1787826302.182898:0:224494:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826302.182920:0:224494:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826302.182962:0:224494:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e87608012f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 224494 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182970:0:224494:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e87608012f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 224494 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826302.182978:0:224494:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e87608012f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 224494 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826302.182996:0:224494:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224494:1874663332573312:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826302.183088:0:224494:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b513c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826302.215290:0:224494:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:224494:1874663332573312:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826302.215323:0:224494:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e87608012f8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a522 expref: -99 pid: 224494 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826303.397278:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332575360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397375:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397403:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397424:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397446:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397473:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:3.0:1787826303.397479:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397502:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397540:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.397574:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.409616:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.409642:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.409655:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.417029:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826303.417262:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332575744:192.168.203.156@tcp:400:kworker.0 00020000:00200000:3.0:1787826303.908878:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826303.908926:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826303.908983:0:136971:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826303.908989:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826303.909013:0:136971:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005e5324a3 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826303.909038:0:136971:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005e5324a3 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826303.909120:0:136971:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826303.909428:0:136971:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb223b80 - object 0x280000400:1228 offset 0<>6 00000100:00100000:3.0:1787826303.909532:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576640:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826303.909670:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826303.909677:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576512:192.168.203.156@tcp:4:bash.0 00000008:00010000:3.0:1787826303.941869:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000005e5324a3 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826303.941924:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000005e5324a3 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826303.941959:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332576512:192.168.203.156@tcp:4:bash.0 00000080:00200000:3.0:1787826303.948759:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826304.708591:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:3.0:1787826306.105174:0:224675:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826306.119162:0:224675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826306.119247:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e8760801330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.119260:0:224675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e8760801330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.119282:0:224675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e8760801330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826306.119308:0:224675:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332577152:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826306.119421:0:224675:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b519c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826306.702660:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826306.719847:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346189312 00000100:00100000:3.0:1787826306.719856:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346189312 00000100:00100000:3.0:1787826306.719909:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43552 00000100:00100000:3.0:1787826306.719918:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8759:x1874663346189312:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826306.719932:0:6158:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826306.719949:0:6158:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826306.719975:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@000000005259c5ee[0x110, 2, [0x240000403:0x1e4b:0x0] hash] 00000020:00010000:3.0:1787826306.719977:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826302 atime: 1787826302 ctime: 1787826302 blocks: 8 00000020:00010000:3.0:1787826306.719980:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@00000000d6f43a36[0x110, 2, [0x280000400:0x4cc:0x0] hash] 00000020:00010000:3.0:1787826306.719982:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826302 atime: 1787826302 ctime: 1787826302 blocks: 8 00000100:00100000:3.0:1787826306.720048:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8759:x1874663346189312:12345-192.168.203.156@tcp:106: Request processed in 126us (411us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826306.720055:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43552 00000100:00100000:3.0:1787826306.720600:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826306.720633:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346189440 00000100:00100000:3.0:1787826306.720638:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346189440 00000100:00100000:3.0:1787826306.720663:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43553 00000100:00100000:3.0:1787826306.720670:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb222300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346189440:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826306.720681:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080134c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x191:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a5fb expref: -99 pid: 224675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.720737:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826306.720745:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb222300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346189440:12345-192.168.203.156@tcp:104: Request processed in 75us (146us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826306.720751:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43553 00000080:00200000:3.0:1787826306.727664:0:224675:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e4b:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826306.727670:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057e166f2e0 (ffff9057f989ce18/0) at cl_lock_release():245 00000020:00010000:3.0:1787826306.727674:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057e166f2e0 (ffff9057f989ce18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.727680:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d1b0 (ffff9057f989c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.727684:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057e166f2e0 (ffff9057f989ce18/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826306.727687:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d1b0 (ffff9057f989ce18/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826306.727723:0:224675:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826306.727831:0:224675:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x240000403:0x1e4b:0x0], ppos: 0, count: 4194304 00000080:00200000:3.0:1787826306.727850:0:224675:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:3.0:1787826306.727866:0:224675:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:3.0:1787826306.727871:0:224675:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826306.727877:0:224675:0:(cl_io.c:214:cl_io_rw_init()) header@000000005fd56bdb[0x110, 1, [0x240000403:0x1e4b:0x0] hash] 00000020:00200000:3.0:1787826306.727879:0:224675:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:3.0:1787826306.727887:0:224675:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826306.727903:0:224675:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:3.0:1787826306.727908:0:224675:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826306.727923:0:224675:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:3.0:1787826306.727937:0:224675:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826306.727942:0:224675:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a718b0, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826306.727985:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.727995:0:224675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.728005:0:224675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826306.728071:0:224675:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332579456:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826306.728140:0:224675:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826306.741333:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346189824 00000100:00100000:3.0:1787826306.741345:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346189824 00000100:00100000:3.0:1787826306.741411:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43554 00000100:00100000:3.0:1787826306.741421:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35352:x1874663346189824:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826306.741437:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000061adc995/0x95194e876080131b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4a576 expref: -99 pid: 224494 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826306.741518:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826306.741533:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35352:x1874663346189824:12345-192.168.203.156@tcp:104: Request processed in 105us (1236us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826306.741540:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43554 00000100:00100000:3.0:1787826306.744756:0:224675:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332579456:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826306.744785:0:224675:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.744810:0:224675:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.744819:0:224675:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826306.751018:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346191104 00000100:00100000:3.0:1787826306.751028:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346191104 00000100:00100000:3.0:1787826306.751094:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43555 00000100:00100000:3.0:1787826306.751104:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6513:x1874663346191104:12345-192.168.203.156@tcp:105: 00010000:00010000:3.0:1787826306.751119:0:22705:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751133:0:22705:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751142:0:22705:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751153:0:22705:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751161:0:22705:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751171:0:22705:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x80000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.751202:0:22705:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9058021f9200) 00000100:00100000:3.0:1787826306.751275:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6513:x1874663346191104:12345-192.168.203.156@tcp:105: Request processed in 172us (425us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826306.751283:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43555 00000100:00100000:3.0:1787826306.752467:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346191232 00000100:00100000:3.0:1787826306.752473:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346191232 00000100:00100000:3.0:1787826306.752510:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43556 00000100:00100000:3.0:1787826306.752519:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346191232:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826306.752540:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801353 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4a68e expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826306.752581:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826306.752588:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346191232:12345-192.168.203.156@tcp:104: Request processed in 71us (195us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826306.752595:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43556 00000020:00010000:3.0:1787826306.763339:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826306.763343:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.763348:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a718b0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826306.763358:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.763372:0:224675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826306.763388:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826306.763391:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a718b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826306.763399:0:224675:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826306.763411:0:224675:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826306.763415:0:224675:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826306.763420:0:224675:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:3.0:1787826306.763438:0:224675:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x240000403:0x1e4b:0x0], ppos: 6, count: 4194304 00000080:00200000:3.0:1787826306.763444:0:224675:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:3.0:1787826306.763448:0:224675:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826306.763452:0:224675:0:(cl_io.c:214:cl_io_rw_init()) header@000000005fd56bdb[0x110, 2, [0x240000403:0x1e4b:0x0] hash] 00000020:00200000:3.0:1787826306.763454:0:224675:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:3.0:1787826306.763458:0:224675:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1787826306.763483:0:224675:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x240000403:0x1e4b:0x0](ffff9057dfdfd348) 00000002:00010000:3.0:1787826306.763491:0:224675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e4b:0x0]) in obj [0x240000403:0x1e4b:0x0], intent: layout flags 0 00010000:00010000:3.0:1787826306.763542:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.763551:0:224675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.763561:0:224675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224675 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826306.763579:0:224675:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332579968:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826306.763660:0:224675:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826306.767432:0:224675:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332579968:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826306.767461:0:224675:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.767478:0:224675:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826306.767493:0:224675:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057fb220380 x1874663332579968/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826322 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826306.767504:0:224675:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826306.767518:0:224675:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.767543:0:224675:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7755 (0 0) 00000080:00200000:3.0:1787826306.767549:0:224675:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x240000403:0x1e4b:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826306.767567:0:224675:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4b:0x0](ffff9057dfdfd348) for lock 0x95194e876080138b 00000080:00010000:3.0:1787826306.767575:0:224675:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080138b from it ffffb512c7323af8 00010000:00010000:3.0:1787826306.767579:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826306.767589:0:224675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767611:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 2/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767622:0:224675:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 2/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826306.767639:0:224675:0:(file.c:6948:ll_layout_lock_set()) ### file [0x240000403:0x1e4b:0x0](0000000008272bc3) being reconfigured ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826306.767677:0:224675:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e4b:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1787826306.767836:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767844:0:224675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1787826306.767868:0:224675:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:3.0:1787826306.767872:0:224675:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826306.767891:0:224675:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:3.0:1787826306.767905:0:224675:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826306.767910:0:224675:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71130, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826306.767925:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767940:0:224675:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767952:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767961:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.767973:0:224675:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826306.767983:0:224675:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:3.0:1787826306.767999:0:224675:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240000403:0x1e4b:0x0] 00000008:00010000:3.0:1787826306.768005:0:224675:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826306.768007:0:224675:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71e30, osc lock ffff9057c50ba850, flags 0x400001002 00010000:00010000:3.0:1787826306.768138:0:224675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.768147:0:224675:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826306.769179:0:224675:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826306.769189:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 7/2,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.769200:0:224675:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1787826306.769208:0:224675:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826306.769216:0:224675:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e4b:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826306.769232:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826306.769236:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.769239:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71e30 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.769243:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826306.769246:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71e30 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826306.769257:0:224675:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:3.0:1787826306.769270:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826306.769273:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826306.769275:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71130 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826306.769280:0:224675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826306.769289:0:224675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826306.769302:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826306.769304:0:224675:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71130 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826306.769310:0:224675:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1787826306.769323:0:224675:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826306.769327:0:224675:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1787826306.769330:0:224675:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000100:00100000:3.0:1787826306.775790:0:224675:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:224675:1874663332580224:192.168.203.156@tcp:35:cat.0 00000002:00100000:3.0:1787826306.775823:0:224675:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158e300 x1874663332577920/t4294985381(4294985381) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826322 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826307.819357:0:224713:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224713:1874663332580352:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826307.819408:0:224713:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819445:0:224713:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819460:0:224713:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819473:0:224713:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819486:0:224713:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819504:0:224713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819513:0:224713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819530:0:224713:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826307.819544:0:224713:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de3b80 x1874663332580352/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826323 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826307.819560:0:224713:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819572:0:224713:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7754 (0 0) 00000080:00200000:3.0:1787826307.819579:0:224713:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826307.819630:0:224713:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e4a:0x0], updating i_size 4096 00000080:00010000:3.0:1787826307.819640:0:224713:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4a:0x0](ffff9057dfdfc2c8) for lock 0x95194e8760801392 00000080:00010000:3.0:1787826307.819647:0:224713:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e4a:0x0](ffff9057dfdfc2c8) 00000080:00010000:3.0:1787826307.819651:0:224713:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801392 from it ffffb512c7413cb8 00010000:00010000:3.0:1787826307.819654:0:224713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819664:0:224713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e8760801392 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a705 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826307.819717:0:224713:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x193:0x0]) in obj [0x240000400:0x193:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826307.819776:0:224713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x193:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224713 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819784:0:224713:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224713 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.819794:0:224713:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224713 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826307.819828:0:224713:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224713:1874663332580480:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826307.819969:0:224713:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057caa26780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826307.835809:0:224713:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224713:1874663332580480:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826307.836129:0:224713:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836146:0:224713:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836161:0:224713:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836176:0:224713:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836190:0:224713:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836198:0:224713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836206:0:224713:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826307.836424:0:224713:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220380 x1874663332580480/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826323 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826307.836444:0:224713:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836460:0:224713:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/403 (0 0) 00010000:00010000:3.0:1787826307.836469:0:224713:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826307.836480:0:224713:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826308.005111:0:6159:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826308.005141:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346193152 00000100:00100000:3.0:1787826308.005154:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346193152 00000100:00100000:3.0:1787826308.005253:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43557 00000100:00100000:3.0:1787826308.005266:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d86df450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346193152:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826308.005289:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801399 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a713 expref: -99 pid: 224713 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826308.005387:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826308.005396:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d86df450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346193152:12345-192.168.203.156@tcp:104: Request processed in 133us (783us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826308.005404:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43557 00000100:00100000:3.0:1787826308.035061:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346194048 00000100:00100000:3.0:1787826308.035076:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346194048 00000100:00100000:3.0:1787826308.035158:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43558 00000100:00100000:3.0:1787826308.035170:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:20167:x1874663346194048:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826308.035194:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801384 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4a6c6 expref: -99 pid: 224675 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826308.035349:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826308.035356:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:20167:x1874663346194048:12345-192.168.203.156@tcp:104: Request processed in 189us (404us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826308.035363:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43558 00000100:00100000:3.0:1787826308.069463:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826308.069503:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826308.069549:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346196352 00000100:00100000:3.0:1787826308.069562:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346196352 00000100:00100000:3.0:1787826308.069618:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346196480 00000100:00100000:3.0:1787826308.069621:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346196480 00000100:00100000:3.0:1787826308.069639:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43559 00000100:00100000:3.0:1787826308.069651:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb220e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346196352:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826308.069674:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080135a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a61e expref: -99 pid: 224677 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826308.069772:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826308.073635:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb220e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346196352:12345-192.168.203.156@tcp:104: Request processed in 130us (330us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826308.073654:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43559 00000100:00100000:3.0:1787826308.085526:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346197248 00000100:00100000:3.0:1787826308.085541:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346197248 00000100:00100000:3.0:1787826308.092450:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826308.274488:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:3.0:1787826308.276707:0:224725:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826308.276733:0:224725:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826308.276750:0:224725:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826308.276765:0:224725:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826308.276773:0:224725:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826308.276780:0:224725:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826308.276783:0:224725:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608013c3 00000080:00010000:3.0:1787826308.276789:0:224725:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608013c3 from it ffffb512c7433be8 00010000:00010000:3.0:1787826308.276793:0:224725:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826308.276802:0:224725:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826308.276815:0:224725:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826308.276823:0:224725:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:3.0:1787826308.276848:0:224725:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826308.276853:0:224725:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826308.276860:0:224725:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826308.276867:0:224725:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826308.276876:0:224725:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826308.276882:0:224725:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826308.276886:0:224725:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826308.276889:0:224725:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608013c3 00000080:00010000:3.0:1787826308.276893:0:224725:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608013c3 from it ffffb512c7433cb8 00010000:00010000:3.0:1787826308.276895:0:224725:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826308.276902:0:224725:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e87608013c3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a801 expref: -99 pid: 224725 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826308.277716:0:224725:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826308.277723:0:224725:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826308.277736:0:224725:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826308.277789:0:224725:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608013d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224725 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826308.277804:0:224725:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608013d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224725 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826308.277814:0:224725:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608013d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224725 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826308.277838:0:224725:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:224725:1874663332584320:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826308.278438:0:224725:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057caa26cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826308.300612:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826308.300715:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826308.457960:0:49149:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_09:398e0287-d3c7-44dd-b1d9-6b85f698e615:49149:1874663332587264:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826308.457990:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080138b lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e4b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4a6f0 expref: -99 pid: 224675 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826308.458028:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826308.458031:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826308.458039:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826308.527340:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332587648:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826308.528068:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332587520:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826309.027136:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332587776:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826309.027281:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787826309.027289:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332587904:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826309.047934:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006cff921b ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801402 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x51f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a8e1 expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826309.047980:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006cff921b ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801402 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x51f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a8e1 expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826309.048008:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332587776:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826309.058800:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973847 gen 19 00000100:00100000:3.0:1787826309.058851:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332587904:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826309.080662:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332588160:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826309.082302:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000003c65c583 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826309.082339:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000003c65c583 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826309.082369:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332588032:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826312.927579:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826312.937899:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826312.939764:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826312.949730:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826312.953593:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:3.0:1787826312.954204:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826312.954212:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954244:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826312.954253:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954262:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954274:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826312.954301:0:49149:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dba:0x0]: losing layout lock 00010000:00010000:3.0:1787826312.954351:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 2/0,0 mode: --/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954361:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826312.954383:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954392:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080143a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dba:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4a9f2 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826312.954419:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826312.954421:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826312.954423:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826312.954439:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332590848:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826312.956929:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826313.535211:0:22705:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826313.535887:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346209920 00000100:00100000:3.0:1787826313.535900:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346209920 00000100:00100000:3.0:1787826313.536047:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43571 00000100:00100000:3.0:1787826313.536060:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d86de850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8443:x1874663346209920:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826313.536082:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e876080146b lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x51f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4aa2a expref: -99 pid: 224895 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826313.536417:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826313.536426:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d86de850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8443:x1874663346209920:12345-192.168.203.156@tcp:104: Request processed in 368us (2691us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826313.536434:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43571 00000100:00100000:3.0:1787826313.634614:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332591744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.634751:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332592000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.634781:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332592256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.634803:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332592512:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.642094:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332591616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.642163:0:2395:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057d128a800: committing for last_committed 4294985390 gen 1 00000100:00100000:3.0:1787826313.642186:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332591872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.642196:0:2395:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957387 gen 1 00000100:00100000:3.0:1787826313.642217:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332592128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826313.642277:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332592384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826314.052500:0:224933:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224933:1874663332592896:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826314.052534:0:224933:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x192:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052548:0:224933:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052561:0:224933:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052574:0:224933:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052588:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052596:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052604:0:224933:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826314.052615:0:224933:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0380 x1874663332592896/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826330 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826314.052628:0:224933:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052648:0:224933:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/402 (0 0) 00010000:00010000:3.0:1787826314.052657:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.052666:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826314.052800:0:224933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826314.052826:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.052837:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.052849:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826314.052856:0:224933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826314.052876:0:224933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826314.052880:0:224933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801417 00000080:00010000:3.0:1787826314.052885:0:224933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801417 from it ffffb512c7613bf8 00010000:00010000:3.0:1787826314.052889:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.052896:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826314.052905:0:224933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826314.052913:0:224933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:3.0:1787826314.052923:0:224933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e4c:0x0](ffff9057dfdfb248), inode mode 41ed mask 44 00000080:00200008:3.0:1787826314.052933:0:224933:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e4c:0x0](ffff9057dfdfb248), flags 704400 00000002:00010000:3.0:1787826314.052952:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e4c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1787826314.053024:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.053032:0:224933:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826314.053046:0:224933:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826314.053053:0:224933:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb221500 x1874663332593152/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826314.053063:0:224933:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826314.053087:0:224933:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224933:1874663332593152:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826314.053338:0:224933:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826314.056306:0:224933:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224933:1874663332593152:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826314.056335:0:224933:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056351:0:224933:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056359:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056367:0:224933:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056377:0:224933:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056384:0:224933:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826314.056390:0:224933:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056397:0:224933:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056405:0:224933:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056413:0:224933:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826314.056420:0:224933:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826314.056427:0:224933:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e876080148e lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 224933 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826314.056459:0:224933:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb221500 x1874663332593152/t4294985396(4294985396) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826330 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826314.056485:0:224933:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb221500 x1874663332593152/t4294985396(4294985396) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826330 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826314.056505:0:224933:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826314.056548:0:224933:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e4c:0x0], updating i_size 4096 00000002:00100000:3.0:1787826314.056572:0:224933:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb221500 x1874663332593152/t4294985396(4294985396) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826330 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826314.056681:0:224933:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e4c:0x0](ffff9057dfdfb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826314.056690:0:224933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e4c:0x0](ffff9057dfdfb248),name=d45j.sanityn 00000002:00010000:3.0:1787826314.056715:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e4c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.056731:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801479 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa69 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056745:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801479 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa69 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826314.056757:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x194:0x0]) in obj [0x240000400:0x194:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.056765:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056773:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056782:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056790:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826314.056805:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x192:0x0]) in obj [0x200000401:0x192:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.056814:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056823:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056831:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.056837:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826314.056848:0:224933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826314.056854:0:224933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e4c:0x0](ffff9057dfdfb248) 00000080:00010000:3.0:1787826314.056858:0:224933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4c:0x0](ffff9057dfdfb248) for lock 0x95194e8760801479 00000080:00010000:3.0:1787826314.057097:0:224933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801479 from it ffffb512c7613cd8 00010000:00010000:3.0:1787826314.057101:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801479 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa69 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.057110:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801479 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa69 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826314.057124:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x194:0x0]) in obj [0x240000400:0x194:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.057130:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057138:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057145:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057152:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:3.0:1787826314.057161:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x192:0x0]) in obj [0x200000401:0x192:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826314.057166:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057175:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057182:0:224933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.057189:0:224933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826314.057343:0:224933:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e4c:0x0](ffff9057dfdfb248) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826314.057359:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:3.0:1787826314.057388:0:224933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x194:0x0]) in obj [0x240000400:0x194:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826314.057394:0:224933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826314.057401:0:224933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787826314.059341:0:224933:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:224933:1874663332593280:192.168.203.156@tcp:37:rm.0 00000100:00100000:3.0:1787826314.059547:0:224933:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826314.168749:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346210432 00000100:00100000:3.0:1787826314.168761:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346210432 00000100:00100000:3.0:1787826314.168822:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43572 00000100:00100000:3.0:1787826314.168834:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346210432:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.168857:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801487 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x192:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4aa85 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826314.168977:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826314.169887:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346210432:12345-192.168.203.156@tcp:104: Request processed in 1051us (1247us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.169898:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43572 00000100:00100000:3.0:1787826314.225596:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346211968 00000100:00100000:3.0:1787826314.225609:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346211968 00000100:00100000:3.0:1787826314.225674:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346212096 00000100:00100000:3.0:1787826314.225677:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346212096 00000100:00100000:3.0:1787826314.225697:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43573 00000100:00100000:3.0:1787826314.225707:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346211968:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.225731:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801441 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a982 expref: -99 pid: 224897 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.225840:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826314.225848:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346211968:12345-192.168.203.156@tcp:104: Request processed in 142us (505us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.225855:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43573 00000100:00100000:3.0:1787826314.225875:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43574 00000100:00100000:3.0:1787826314.225882:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346212096:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.225892:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a919 expref: -99 pid: 224895 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826314.225932:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826314.225938:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346212096:12345-192.168.203.156@tcp:104: Request processed in 57us (532us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.225944:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43574 00000100:00100000:3.0:1787826314.252154:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346212864 00000100:00100000:3.0:1787826314.252167:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346212864 00000100:00100000:3.0:1787826314.252271:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43575 00000100:00100000:3.0:1787826314.252283:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346212864:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.252306:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801448 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4a99e expref: -99 pid: 224897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.252465:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826314.252482:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346212864:12345-192.168.203.156@tcp:104: Request processed in 200us (505us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.252491:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43575 00000100:00100000:3.0:1787826314.259218:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346213120 00000100:00100000:3.0:1787826314.259280:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346213120 00000100:00100000:3.0:1787826314.259348:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43576 00000100:00100000:3.0:1787826314.259361:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35093:x1874663346213120:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.259385:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801480 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4aa77 expref: -99 pid: 224933 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826314.259979:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826314.259997:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35093:x1874663346213120:12345-192.168.203.156@tcp:104: Request processed in 630us (871us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.260010:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43576 00000100:00100000:3.0:1787826314.423899:0:6159:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826314.423933:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346214784 00000100:00100000:3.0:1787826314.423945:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346214784 00000100:00100000:3.0:1787826314.424028:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346214912 00000100:00100000:3.0:1787826314.424033:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346214912 00000100:00100000:3.0:1787826314.424061:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43577 00000100:00100000:3.0:1787826314.424072:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346214784:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.424096:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608014b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ab81 expref: -99 pid: 224945 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.424197:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826314.424203:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346214784:12345-192.168.203.156@tcp:104: Request processed in 133us (3842us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.424211:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43577 00000100:00100000:3.0:1787826314.424259:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43578 00000100:00100000:3.0:1787826314.424266:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c6d89c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346214912:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826314.424276:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2a00/0x95194e87608014aa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ab5e expref: -99 pid: 224945 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826314.424309:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826314.424315:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c6d89c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346214912:12345-192.168.203.156@tcp:104: Request processed in 50us (3884us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826314.424321:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43578 00010000:00010000:3.0:1787826314.549608:0:224950:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826314.549625:0:224950:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1787826314.549650:0:224947:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826314.549662:0:224947:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/7 locks to regular blp list, 3 blwis in pool 00010000:00010000:3.0:1787826314.549686:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 4 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1787826314.549694:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2cc00/0x95194e87608014bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4abc7 expref: -99 pid: 224934 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.549728:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c600/0x95194e87608014e2 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4ac29 expref: -99 pid: 224934 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826314.549741:0:6167:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbb:0x0]: losing layout lock 00010000:00010000:3.0:1787826314.549761:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d200/0x95194e87608014c6 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4abe3 expref: -99 pid: 224934 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.549796:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2cc00/0x95194e87608014bf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4abc7 expref: -99 pid: 224934 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.549803:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c600/0x95194e87608014e2 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ac29 expref: -99 pid: 224934 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.549810:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d200/0x95194e87608014c6 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4abe3 expref: -99 pid: 224934 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.549815:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:3.0:1787826314.549837:0:6167:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:fc0b407b-a541-420d-8712-b9c8da18fe8c:6167:1874663332599040:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826314.549904:0:6167:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00010000:00010000:3.0:1787826314.549918:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826314.549923:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801472 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dba:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4aa46 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826314.549940:0:34879:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dba:0x0]: losing layout lock 00010000:00010000:3.0:1787826314.549957:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801472 lrc: 1/0,0 mode: --/CR res: [0x200000403:0x1dba:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4aa46 expref: -99 pid: 224895 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826314.549963:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826314.549970:0:34879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_06:398e0287-d3c7-44dd-b1d9-6b85f698e615:34879:1874663332599168:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826314.549991:0:34879:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826314.560836:0:15672:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:fc0b407b-a541-420d-8712-b9c8da18fe8c:15672:1874663332599296:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826314.560862:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2d600/0x95194e87608014f0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ac53 expref: -99 pid: 224934 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.560888:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2c000/0x95194e87608014cd lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4abf1 expref: -99 pid: 224934 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.560900:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080144f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e4c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4a9ac expref: -99 pid: 224897 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826314.560923:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826314.560925:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826314.560928:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826315.170460:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826315.170491:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826315.170522:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826318.475936:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599424:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826318.475982:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.475999:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476013:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476025:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476040:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476049:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476057:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826318.476078:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de3480 x1874663332599424/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826318.476093:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476109:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826318.476117:0:225115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826318.476156:0:225115:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826318.476163:0:225115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608014fe 00000080:00010000:3.0:1787826318.476169:0:225115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826318.476172:0:225115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608014fe from it ffffb512c7783bf8 00010000:00010000:3.0:1787826318.476176:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476186:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826318.476216:0:225115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826318.476473:0:225115:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=25 00000080:00200000:3.0:1787826318.476480:0:225115:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=0 00000002:00010000:3.0:1787826318.476508:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826318.476562:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476576:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.476587:0:225115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826318.476611:0:225115:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599552:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826318.476853:0:225115:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826318.482146:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599552:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826318.482182:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482215:0:225115:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482520:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482533:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482546:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482562:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482572:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482580:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826318.482594:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb223b80 x1874663332599552/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826318.482607:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482623:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7758 (0 0) 00000002:00010000:3.0:1787826318.482645:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e4e:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826318.482706:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482723:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.482733:0:225115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826318.482918:0:225115:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599680:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826318.483130:0:225115:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826318.495329:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599680:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826318.495368:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.495383:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.495400:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.495412:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496000:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496011:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496018:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826318.496031:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb222300 x1874663332599680/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826318.496045:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496059:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7758 (0 0) 00000080:00200000:3.0:1787826318.496083:0:225115:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826318.496135:0:225115:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448) 00000002:00010000:3.0:1787826318.496172:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x195:0x0]) in obj [0x240000400:0x195:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826318.496295:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496305:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.496315:0:225115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826318.496338:0:225115:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599808:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826318.496451:0:225115:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826318.500012:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599808:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826318.500046:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500060:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500073:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500086:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500101:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500110:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500117:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826318.500130:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223b80 x1874663332599808/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826318.500144:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500158:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/405 (0 0) 00010000:00010000:3.0:1787826318.500168:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.500177:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826318.500213:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x193:0x0]) in obj [0x200000401:0x193:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826318.503819:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.503840:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.504084:0:225115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826318.504119:0:225115:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599936:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826318.504338:0:225115:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826318.508793:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332599936:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826318.508828:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508843:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508856:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508867:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508882:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508891:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508899:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826318.508912:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223b80 x1874663332599936/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826318.508926:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508941:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/403 (0 0) 00010000:00010000:3.0:1787826318.508950:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.508959:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826318.509012:0:225115:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e4e:0x0], updating i_size 8192 00000080:00010000:3.0:1787826318.509025:0:225115:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4e:0x0](ffff9057dfdef448) for remote lock 0x95194e876080150c 00000080:00010000:3.0:1787826318.509031:0:225115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4e:0x0](ffff9057dfdef448) for lock 0x95194e8760801505 00000080:00010000:3.0:1787826318.509044:0:225115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e4e:0x0](ffff9057dfdef448) 00000080:00010000:3.0:1787826318.509049:0:225115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801505 from it ffffb512c7783ac8 00010000:00010000:3.0:1787826318.509053:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.509061:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e8760801505 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac8b expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826318.509069:0:225115:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e876080150c from it ffffb512c7783ac8 00010000:00010000:3.0:1787826318.509073:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.509081:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e876080150c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac99 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826318.509107:0:225115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:3.0:1787826318.509134:0:225115:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), file ffff9057d0e36c00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1787826318.509141:0:225115:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), intent=open 00000002:00010000:3.0:1787826318.509166:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x193:0x0], intent: open flags 0100001 00010000:00010000:3.0:1787826318.509462:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826318.509472:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826318.509485:0:225115:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826318.509492:0:225115:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222300 x1874663332600192/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826318.509502:0:225115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826318.509530:0:225115:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332600192:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826318.510164:0:225115:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d900 going to sleep for 11 seconds 00000100:00100000:3.0:1787826318.760370:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332600320:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826318.760398:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332600704:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826318.760410:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332600960:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826318.760422:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332600448:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826318.760434:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332600832:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826318.760457:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332601088:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826319.031893:0:225115:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225115:1874663332600192:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826319.031931:0:225115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.031971:0:225115:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.031980:0:225115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.031993:0:225115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032009:0:225115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032024:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032032:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032038:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826319.032051:0:225115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb222300 x1874663332600192/t8589957395(8589957395) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826334 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826319.032065:0:225115:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb222300 x1874663332600192/t8589957395(8589957395) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826334 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826319.032078:0:225115:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826319.032091:0:225115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032103:0:225115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935619/7611 (0 0) 00000080:00200000:3.0:1787826319.032110:0:225115:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000401:0x193:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826319.032385:0:225115:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dbb:0x0](ffff9057dfdea1c8) 00000080:00200000:3.0:1787826319.032505:0:225115:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbb:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826319.032512:0:225115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbb:0x0](ffff9057dfdea1c8) for lock 0x95194e8760801521 00000080:00010000:3.0:1787826319.032528:0:225115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbb:0x0](ffff9057dfdea1c8) 00000080:00200008:3.0:1787826319.032547:0:225115:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1dbb:0x0](ffff9057dfdea1c8), flags 100000 00000002:00100000:3.0:1787826319.032576:0:225115:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb222300 x1874663332600192/t8589957395(8589957395) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826334 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826319.032639:0:225115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801521 from it ffff9057ef265720 00010000:00010000:3.0:1787826319.032644:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.032655:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826319.032681:0:225115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1dbb:0x0](ffff9057dfdea1c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826319.032726:0:225115:0:(file.c:6296:ll_getattr_dentry()) START file f45j.sanityn:[0x200000403:0x1dbb:0x0](ffff9057dfdea1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826319.032733:0:225115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000403:0x1dbb:0x0](ffff9057dfdea1c8),name=f45j.sanityn 00000002:00010000:3.0:1787826319.032755:0:225115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dbb:0x0]) in obj [0x200000401:0x193:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826319.032765:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 2/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.032778:0:225115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 2/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826319.032786:0:225115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x193:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826319.032791:0:225115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbb:0x0](ffff9057dfdea1c8) 00000080:00010000:3.0:1787826319.032795:0:225115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbb:0x0](ffff9057dfdea1c8) for lock 0x95194e8760801521 00000080:00010000:3.0:1787826319.032798:0:225115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801521 from it ffffb512c7783cd8 00010000:00010000:3.0:1787826319.032802:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.032809:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801521 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ad48 expref: -99 pid: 225115 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826319.032828:0:225115:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbb:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826319.033184:0:225115:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbb:0x0] 00000008:00010000:3.0:1787826319.033205:0:225115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826319.033210:0:225115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a718b0, osc lock ffff9057c50ba390, flags 0x400001002 00000100:00080000:3.0:1787826319.033822:0:225115:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8097 00000100:00080000:3.0:1787826319.034663:0:225115:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826319.034679:0:225115:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8437760, lost: 0. 00000100:00100000:3.0:1787826319.035554:0:225115:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332602240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826319.035645:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000ca879f03/0x95194e876080154b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826319.035660:0:225115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000ca879f03/0x95194e876080154b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826319.035718:0:225115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787826319.035743:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1787826319.035763:0:2396:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9057fb223b80 x1874663332602368/t0(0) o101->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826319.042638:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346220416 00000100:00100000:3.0:1787826319.042652:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346220416 00000100:00100000:3.0:1787826319.042742:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43579 00000100:00100000:3.0:1787826319.042755:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346220416:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826319.042778:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801513 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4aca7 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.042877:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826319.042886:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346220416:12345-192.168.203.156@tcp:104: Request processed in 133us (1075us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826319.042894:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43579 00000100:00100000:3.0:1787826319.058779:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43581 00000100:00100000:3.0:1787826319.058792:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346221440:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826319.058807:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.058862:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826319.058868:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346221440:12345-192.168.203.156@tcp:104: Request processed in 81us (1788us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826319.058875:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43581 00010000:00010000:3.0:1787826319.058897:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1787826319.058911:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.058921:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2400 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:3.0:1787826319.058927:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400010000 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826319.058969:0:49149:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826319.059051:0:49149:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826319.059113:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 2/0,0 mode: --/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.059635:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826319.059667:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 1/0,0 mode: --/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.059676:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000a3e2cf53/0x95194e8760801410 lrc: 0/0,0 mode: --/PW res: [0x2c0000400:0x4ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4a90b expref: -99 pid: 224714 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.059706:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826319.059710:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059718:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826319.059724:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059733:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059742:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059793:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 2/0,0 mode: --/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059801:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826319.059807:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 1/0,0 mode: --/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059814:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e876080151a lrc: 0/0,0 mode: --/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4acb5 expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.059830:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826319.059832:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826319.059834:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826319.059853:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332602752:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826319.059922:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826319.059929:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332602624:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826319.064172:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332602624:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826319.064194:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332602752:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1787826319.070042:0:225115:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbb:0x0] updating i_size 6 i_blocks 1 00000020:00010000:3.0:1787826319.070050:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826319.070053:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.070060:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a718b0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.070065:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826319.070067:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a718b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826319.070095:0:225115:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbb:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826319.072720:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826319.075964:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346222720 00000100:00100000:3.0:1787826319.075976:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346222720 00000080:00200000:3.0:1787826319.110093:0:15672:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787826319.110183:0:15672:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787826319.112532:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e87608014e9 lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4ac37 expref: -99 pid: 224934 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.112554:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826319.112665:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e87608014e9 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4ac37 expref: -99 pid: 224934 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.112676:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e87608014e9 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4ac37 expref: -99 pid: 224934 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.112701:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826319.112704:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826319.112706:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826319.112714:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826319.112735:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332603264:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826319.116133:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826319.131353:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332603264:192.168.203.156@tcp:103:ldlm_bl.0 00000020:00010000:3.0:1787826319.146171:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c5c8d218 (ffff905722debe18/0) at cl_lock_release():245 00000020:00010000:3.0:1787826319.146175:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c5c8d218 (ffff905722debe18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.146180:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5dbb0 (ffff9057d889c638/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826319.146190:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826319.146206:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826319.146268:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c5c8d218 (ffff905722debe18/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826319.146272:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5dbb0 (ffff905722debe18/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826319.146281:0:225115:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826319.146356:0:225115:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826319.146361:0:225115:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826319.146366:0:225115:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:3.0:1787826319.147769:0:225115:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbb:0x0], ppos: 6, count: 4194304 00000080:00200000:3.0:1787826319.147787:0:225115:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:3.0:1787826319.147791:0:225115:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826319.147798:0:225115:0:(cl_io.c:214:cl_io_rw_init()) header@00000000fc32c953[0x110, 2, [0x200000403:0x1dbb:0x0] hash] 00000020:00200000:3.0:1787826319.147800:0:225115:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:3.0:1787826319.147805:0:225115:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826319.147823:0:225115:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:3.0:1787826319.147827:0:225115:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826319.147844:0:225115:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:3.0:1787826319.147861:0:225115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826319.147867:0:225115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a718b0, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826319.147888:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.147903:0:225115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.147918:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.147929:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.147937:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826319.147945:0:225115:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:3.0:1787826319.147960:0:225115:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbb:0x0] 00000008:00010000:3.0:1787826319.147966:0:225115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826319.147968:0:225115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71330, osc lock ffff9057c50ba850, flags 0x400001002 00010000:00010000:3.0:1787826319.147975:0:225115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.147983:0:225115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826319.147992:0:225115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826319.147995:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.148003:0:225115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826319.148010:0:225115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826319.148016:0:225115:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbb:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826319.148019:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826319.148023:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.148025:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71330 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.148029:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826319.148032:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71330 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826319.148038:0:225115:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:3.0:1787826319.148048:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826319.148050:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826319.148052:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a718b0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826319.148057:0:225115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826319.148066:0:225115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:3.0:1787826319.148078:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826319.148081:0:225115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a718b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826319.148086:0:225115:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1787826319.148095:0:225115:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826319.148098:0:225115:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1787826319.148101:0:225115:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000100:00100000:3.0:1787826319.560970:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346226560 00000100:00100000:3.0:1787826319.560985:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346226560 00000100:00100000:3.0:1787826319.561294:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43586 00000100:00100000:3.0:1787826319.561306:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8443:x1874663346226560:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826319.561323:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000076308332/0x95194e8760801552 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4ad95 expref: -99 pid: 225115 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826319.561414:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826319.561421:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8443:x1874663346226560:12345-192.168.203.156@tcp:104: Request processed in 118us (2955us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826319.561428:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43586 00000008:00010000:3.0:1787826319.788673:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000000ed20816 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000935242d3/0x95194e87608014f7 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ac61 expref: -99 pid: 224934 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826319.788726:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000000ed20816 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000935242d3/0x95194e87608014f7 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ac61 expref: -99 pid: 224934 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826319.788763:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332604032:192.168.203.156@tcp:4:bash.0 00000080:00200000:3.0:1787826320.390926:0:225153:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826320.391048:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.391070:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826320.391087:0:225153:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826320.391095:0:225153:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826320.391103:0:225153:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826320.391107:0:225153:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608014fe 00000080:00010000:3.0:1787826320.391184:0:225153:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608014fe from it ffffb512c77e3b58 00010000:00010000:3.0:1787826320.391189:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826320.391199:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826320.391358:0:225153:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826320.391368:0:225153:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:3.0:1787826320.391377:0:225153:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e4e:0x0](ffff9057dfdef448), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826320.391381:0:225153:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448),name=d45j.sanityn 00000002:00010000:3.0:1787826320.391395:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e4e:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.391403:0:225153:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7758 (0 0) 00010000:00010000:3.0:1787826320.391464:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.391506:0:225153:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.391517:0:225153:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.391543:0:225153:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604288:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826320.391683:0:225153:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826320.398657:0:225153:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057d128a800: committing for last_committed 4294985401 gen 1 00000100:00100000:3.0:1787826320.398687:0:225153:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604288:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826320.398722:0:225153:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398745:0:225153:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398754:0:225153:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398765:0:225153:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398777:0:225153:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398791:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398799:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398806:0:225153:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826320.398819:0:225153:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220000 x1874663332604288/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826336 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826320.398832:0:225153:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.398842:0:225153:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7758 (0 0) 00000080:00200000:3.0:1787826320.398849:0:225153:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826320.398877:0:225153:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e4e:0x0], updating i_size 4096 00000080:00010000:3.0:1787826320.399837:0:225153:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e4e:0x0](ffff9057dfdef448) for lock 0x95194e8760801560 00000080:00010000:3.0:1787826320.399851:0:225153:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e4e:0x0](ffff9057dfdef448) 00000080:00010000:3.0:1787826320.399854:0:225153:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801560 from it ffffb512c77e3cb8 00010000:00010000:3.0:1787826320.399860:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.399872:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e8760801560 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4adcd expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826320.400039:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x195:0x0]) in obj [0x240000400:0x195:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.400090:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.400099:0:225153:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.400107:0:225153:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.401809:0:225153:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604416:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826320.401939:0:225153:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826320.407510:0:225153:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604416:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826320.407537:0:225153:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407550:0:225153:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407560:0:225153:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407571:0:225153:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407584:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407592:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407600:0:225153:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826320.407612:0:225153:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220000 x1874663332604416/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826336 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826320.407625:0:225153:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407637:0:225153:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/405 (0 0) 00010000:00010000:3.0:1787826320.407644:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407653:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826320.407678:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x193:0x0]) in obj [0x200000401:0x193:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.407733:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407741:0:225153:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.407750:0:225153:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.407768:0:225153:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604544:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826320.407840:0:225153:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826320.418649:0:225153:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057d128a800: committing for last_committed 8589957399 gen 1 00000100:00100000:3.0:1787826320.418666:0:225153:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332604544:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826320.418688:0:225153:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 4/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418701:0:225153:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 4/1,0 mode: --/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418713:0:225153:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418724:0:225153:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418735:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418743:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418751:0:225153:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826320.418763:0:225153:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220000 x1874663332604544/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826336 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826320.418777:0:225153:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418789:0:225153:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/403 (0 0) 00010000:00010000:3.0:1787826320.418796:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.418806:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826320.418877:0:225153:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826320.418904:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.418914:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826320.418925:0:225153:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826320.418933:0:225153:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826320.418939:0:225153:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826320.418942:0:225153:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608014fe 00000080:00010000:3.0:1787826320.418947:0:225153:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608014fe from it ffffb512c77e3bf8 00010000:00010000:3.0:1787826320.419593:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826320.419613:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826320.419625:0:225153:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826320.419633:0:225153:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000100:00100000:3.0:1787826320.509291:0:225153:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239880 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332605696:192.168.203.156@tcp:41:rm.0 00800000:00000004:3.0:1787826320.509320:0:225153:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793371/1916558 objects 519501/534196 00000080:00000004:3.0:1787826320.509329:0:225153:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793371/1916558 objects 519501/534196 00000080:00000004:3.0:1787826320.509334:0:225153:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793371/1916558 objects 519501/534196 00000080:00200000:3.0:1787826320.509369:0:225153:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826320.509399:0:225153:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608014fe lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ac6f expref: -99 pid: 225115 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:3.0:1787826320.509455:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x195:0x0]) in obj [0x240000400:0x195:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826320.509468:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826320.509480:0:225153:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:3.0:1787826320.509494:0:225153:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:3.0:1787826320.509499:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826320.509508:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801567 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4addb expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:3.0:1787826320.509544:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x193:0x0]) in obj [0x200000401:0x193:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826320.509551:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826320.509560:0:225153:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:3.0:1787826320.509571:0:225153:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:3.0:1787826320.509575:0:225153:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826320.509582:0:225153:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1787826320.509621:0:225153:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:3.0:1787826320.509643:0:225153:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:3.0:1787826320.509657:0:225153:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), flags=0 00000080:00200000:3.0:1787826320.509661:0:225153:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e4e:0x0](ffff9057dfdef448), intent=getattr 00000002:00010000:3.0:1787826320.509675:0:225153:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x193:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826320.509841:0:225153:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801583 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.509850:0:225153:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801583 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.509859:0:225153:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801583 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225153 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.509882:0:225153:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332605824:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826320.510086:0:225153:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826320.532377:0:225153:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225153:1874663332605824:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826320.532404:0:225153:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801583 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x193:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ae28 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.548855:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346227200 00000100:00100000:3.0:1787826320.548866:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346227200 00000100:00100000:3.0:1787826320.548926:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43587 00000100:00100000:3.0:1787826320.548937:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346227200:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826320.549090:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e876080156e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x193:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ade9 expref: -99 pid: 225153 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826320.549241:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826320.549249:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346227200:12345-192.168.203.156@tcp:104: Request processed in 314us (883us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826320.549257:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43587 00000100:00100000:3.0:1787826320.648349:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826320.658180:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826320.833671:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826320.833743:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826320.841832:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332608640:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826320.846152:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332608640:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1787826320.874280:0:225154:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826320.874307:0:225154:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826320.874366:0:225154:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.874377:0:225154:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.874387:0:225154:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.874411:0:225154:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225154:1874663332608768:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826320.874896:0:225154:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826320.882133:0:225154:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225154:1874663332608768:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826320.882162:0:225154:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882175:0:225154:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882187:0:225154:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882198:0:225154:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882212:0:225154:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882372:0:225154:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882381:0:225154:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826320.882394:0:225154:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222d80 x1874663332608768/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826336 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826320.882408:0:225154:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882419:0:225154:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826320.882436:0:225154:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826320.882466:0:225154:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826320.882472:0:225154:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608015a6 00000080:00010000:3.0:1787826320.882477:0:225154:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826320.882481:0:225154:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608015a6 from it ffffb512c77f3bf8 00010000:00010000:3.0:1787826320.882485:0:225154:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882494:0:225154:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608015a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4af2b expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826320.882519:0:225154:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826320.882536:0:225154:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:3.0:1787826320.882541:0:225154:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826320.882559:0:225154:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826320.882596:0:225154:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608015ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882603:0:225154:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608015ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826320.882611:0:225154:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608015ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225154 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826320.882631:0:225154:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225154:1874663332608896:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826320.882713:0:225154:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00010000:00010000:3.0:1787826320.998559:0:225167:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826320.998597:0:225167:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787826320.998643:0:225169:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826320.998885:0:225169:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/7 locks to regular blp list, 3 blwis in pool 00010000:00010000:3.0:1787826320.998918:0:225170:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00000100:00100000:3.0:1787826321.008910:0:49150:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_10:fc0b407b-a541-420d-8712-b9c8da18fe8c:49150:1874663332610944:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826321.008925:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e87608015d7 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4afb7 expref: -99 pid: 225154 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826321.008940:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2c800/0x95194e87608015b4 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4af55 expref: -99 pid: 225154 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826321.008951:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801536 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e4e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ad02 expref: -99 pid: 225117 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826321.008974:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826321.008976:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826321.008978:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826323.874617:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332611072:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874777:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611328:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874808:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611584:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874842:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611840:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874876:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332612096:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874907:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:3.0:1787826323.874913:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611200:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874933:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611456:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874956:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611712:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.874976:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611968:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.879733:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332611072:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.880120:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611200:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.880134:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611456:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.880148:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611712:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.880162:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611968:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.880174:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332612096:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.884380:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332611328:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826323.888589:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332611840:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826324.679563:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:3.0:1787826324.898436:0:136971:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826324.898458:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826324.898488:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826324.898504:0:136971:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826324.898509:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826324.898531:0:136971:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000066f642da ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826324.898556:0:136971:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000066f642da ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826324.898638:0:136971:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826324.898755:0:136971:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb223100 - object 0x280000401:1312 offset 0<>6 00000100:00100000:3.0:1787826324.901562:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332612480:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826324.906625:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000066f642da ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826324.906681:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000066f642da ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826324.906718:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332612352:192.168.203.156@tcp:4:bash.0 00000080:00200000:3.0:1787826324.914805:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e51:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826325.040883:0:225335:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826325.040926:0:225335:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826325.041793:0:225335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.041810:0:225335:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.041822:0:225335:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826325.041855:0:225335:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225335:1874663332612864:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826325.042542:0:225335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826325.048981:0:225335:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225335:1874663332612864:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826325.049018:0:225335:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049032:0:225335:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049047:0:225335:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049059:0:225335:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049075:0:225335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049085:0:225335:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049094:0:225335:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826325.049108:0:225335:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220380 x1874663332612864/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826325.049124:0:225335:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049138:0:225335:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826325.049146:0:225335:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826325.049183:0:225335:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826325.049190:0:225335:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608015e5 00000080:00010000:3.0:1787826325.049196:0:225335:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826325.049199:0:225335:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608015e5 from it ffffb512c79abbf8 00010000:00010000:3.0:1787826325.049203:0:225335:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049213:0:225335:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826325.049277:0:225335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826325.049302:0:225335:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=25 00000080:00200000:3.0:1787826325.049308:0:225335:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=0 00000002:00010000:3.0:1787826325.049335:0:225335:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826325.049630:0:225335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049639:0:225335:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.049649:0:225335:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225335 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826325.049687:0:225335:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225335:1874663332612992:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826325.049822:0:225335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826325.075564:0:225335:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225335:1874663332612992:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826325.075594:0:225335:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075619:0:225335:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075628:0:225335:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075640:0:225335:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075652:0:225335:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075665:0:225335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075675:0:225335:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075696:0:225335:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826325.075712:0:225335:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb220380 x1874663332612992/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826325.075729:0:225335:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e87608015ec lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4aff6 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.075740:0:225335:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7760 (0 0) 00000080:00200000:3.0:1787826325.413773:0:136971:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826325.413796:0:136971:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826325.413833:0:136971:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826325.631866:0:225337:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225337:1874663332614272:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826325.631919:0:225337:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: --/CR res: [0x200000401:0x194:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.631964:0:225337:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.631974:0:225337:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.631994:0:225337:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632006:0:225337:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632021:0:225337:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632029:0:225337:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632036:0:225337:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826325.632049:0:225337:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0e00 x1874663332614272/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826325.632060:0:225337:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826325.632075:0:225337:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632088:0:225337:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7612 (0 0) 00000080:00200000:3.0:1787826325.632096:0:225337:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x194:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826325.632154:0:225337:0:(namei.c:147:ll_iget()) got inode: [0x200000403:0x1dbc:0x0](ffff9057dfe05b88): rc = 0 00000080:00200000:3.0:1787826325.632170:0:225337:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826325.632175:0:225337:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbc:0x0](ffff9057dfe05b88) for lock 0x95194e876080162b 00000080:00010000:3.0:1787826325.632187:0:225337:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbc:0x0](ffff9057dfe05b88) 00000080:00010000:3.0:1787826325.632191:0:225337:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080162b from it ffffb512c79c3db8 00010000:00010000:3.0:1787826325.632195:0:225337:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.632204:0:225337:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826325.632274:0:225337:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e50:0x0](ffff9057dfdf4b08), inode mode 41ed mask 3 00000080:00200000:3.0:1787826325.632281:0:225337:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e50:0x0](ffff9057dfdf4b08), inode mode 41ed mask 3 00000080:00200000:3.0:1787826325.632291:0:225337:0:(namei.c:2492:ll_rename()) VFS Op:oldname=f45j.sanityn-2, src_dir=[0x240000403:0x1e50:0x0](ffff9057dfdf4b08), newname=f45j.sanityn, tgt_dir=[0x240000403:0x1e50:0x0](ffff9057dfdf4b08) 00010000:00010000:3.0:1787826325.632332:0:225337:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801624 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e51:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b089 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826325.632350:0:225337:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e51:0x0]: losing layout lock 00010000:00010000:3.0:1787826325.632409:0:225337:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801624 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e51:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b089 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826325.632416:0:225337:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826325.632451:0:225337:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801624 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e51:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b089 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826325.632477:0:225337:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826325.632487:0:225337:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbc:0x0]: losing layout lock 00010000:00010000:3.0:1787826325.632507:0:225337:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826325.632514:0:225337:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826325.632516:0:225337:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06000/0x95194e876080162b lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b0a5 expref: -99 pid: 225337 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1787826325.632541:0:225337:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826325.632548:0:225337:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222300 x1874663332614656/t0(0) o36->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:13/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826325.632570:0:225337:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225337:1874663332614656:192.168.203.156@tcp:36:mrename.0 00000100:00100000:3.0:1787826325.632868:0:225337:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826325.655471:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826325.672803:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346238848 00000100:00100000:3.0:1787826325.672817:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346238848 00000100:00100000:3.0:1787826325.672891:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43595 00000100:00100000:3.0:1787826325.672904:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346238848:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826325.672923:0:6158:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826325.672938:0:6158:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826325.672970:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@000000000f4e072b[0x110, 2, [0x200000403:0x1dbc:0x0] hash] 00000020:00010000:3.0:1787826325.672973:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826320 atime: 1787826320 ctime: 1787826320 blocks: 8 00000020:00010000:3.0:1787826325.672977:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@00000000885cee9b[0x110, 2, [0x280000401:0x520:0x0] hash] 00000020:00010000:3.0:1787826325.672979:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826320 atime: 1787826320 ctime: 1787826320 blocks: 8 00000100:00100000:3.0:1787826325.673045:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346238848:12345-192.168.203.156@tcp:106: Request processed in 143us (502us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826325.673053:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43595 00000100:00100000:3.0:1787826325.676266:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346239232 00000100:00100000:3.0:1787826325.676278:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346239232 00000100:00100000:3.0:1787826325.676345:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43596 00000100:00100000:3.0:1787826325.676357:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346239232:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826325.676381:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801601 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b020 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826325.676495:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826325.676503:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346239232:12345-192.168.203.156@tcp:104: Request processed in 148us (454us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826325.676511:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43596 00000080:00200000:3.0:1787826325.678639:0:225335:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbc:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826325.678647:0:225335:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9056c416d720 (ffff905722deb518/0) at cl_lock_release():245 00000020:00010000:3.0:1787826325.678661:0:225335:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9056c416d720 (ffff905722deb518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826325.678668:0:225335:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d630 (ffff9057d889cab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826325.678673:0:225335:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9056c416d720 (ffff905722deb518/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826325.678676:0:225335:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d630 (ffff905722deb518/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826325.678726:0:225335:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826325.678881:0:225335:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbc:0x0], ppos: 0, count: 4194304 00000080:00200000:3.0:1787826325.678908:0:225335:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:3.0:1787826325.678941:0:225335:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:3.0:1787826325.678947:0:225335:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826325.678955:0:225335:0:(cl_io.c:214:cl_io_rw_init()) header@00000000234a4055[0x110, 1, [0x200000403:0x1dbc:0x0] hash] 00000020:00200000:3.0:1787826325.678957:0:225335:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:3.0:1787826325.678962:0:225335:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826325.678981:0:225335:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:3.0:1787826325.678985:0:225335:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826325.679003:0:225335:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:3.0:1787826325.679016:0:225335:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826325.679021:0:225335:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71cb0, osc lock ffff9057c50ba390, flags 0x0 00010000:00010000:3.0:1787826325.679075:0:225335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801639 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225335 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.679085:0:225335:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801639 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225335 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826325.679112:0:225335:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801639 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225335 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826325.679136:0:225335:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225335:1874663332615168:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826325.679355:0:225335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826325.697007:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346239872 00000100:00100000:3.0:1787826325.697020:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346239872 00000100:00100000:3.0:1787826325.697103:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43597 00000100:00100000:3.0:1787826325.697116:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8445:x1874663346239872:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826325.697139:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e87608015d0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4af9b expref: -99 pid: 225154 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826325.697269:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826325.697277:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8445:x1874663346239872:12345-192.168.203.156@tcp:104: Request processed in 164us (403us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826325.697284:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43597 00000100:00100000:3.0:1787826325.710981:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826325.712122:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826325.716157:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346240768 00000100:00100000:3.0:1787826325.716178:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346240768 00000100:00100000:3.0:1787826325.716277:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43599 00000100:00100000:3.0:1787826325.716292:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d86dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346240896:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826325.716306:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801608 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4b0ba expref: -99 pid: 225335 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826325.716427:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826325.716436:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d86dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346240896:12345-192.168.203.156@tcp:104: Request processed in 149us (4318us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826325.716444:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43599 00000100:00100000:3.0:1787826325.738351:0:225337:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225337:1874663332614656:192.168.203.156@tcp:36:mrename.0 00000100:00100000:3.0:1787826326.094826:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346243072 00000100:00100000:3.0:1787826326.094841:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346243072 00000100:00100000:3.0:1787826326.094925:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43600 00000100:00100000:3.0:1787826326.094938:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346243072:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826326.094964:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e8760801639 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x520:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4b0eb expref: -99 pid: 225335 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826326.095073:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826326.095080:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346243072:12345-192.168.203.156@tcp:104: Request processed in 145us (665us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826326.095087:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43600 00010000:00010000:3.0:1787826326.831273:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e876080165c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225373 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826326.831283:0:225373:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e876080165c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225373 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826326.831296:0:225373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826326.831305:0:225373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222a00 x1874663332616704/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826326.831315:0:225373:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e876080165c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225373 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826326.831341:0:225373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332616704:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826326.831584:0:225373:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000002:00010000:0.0:1787826326.835638:0:225373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x194:0x0]) in obj [0x200000401:0x194:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826326.835648:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.835657:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.835664:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.835671:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826326.835683:0:225373:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826326.835695:0:225373:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e50:0x0](ffff9057dfdf5b88) 00000080:00010000:0.0:1787826326.835699:0:225373:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e50:0x0](ffff9057dfdf5b88) for lock 0x95194e8760801647 00000080:00010000:0.0:1787826326.835704:0:225373:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801647 from it ffffb512c7a1bcd8 00010000:00010000:0.0:1787826326.835707:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07000/0x95194e8760801647 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b12a expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.835714:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07000/0x95194e8760801647 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b12a expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826326.835791:0:225373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x196:0x0]) in obj [0x240000400:0x196:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826326.835800:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.835811:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.835827:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.836044:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826326.836056:0:225373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x194:0x0]) in obj [0x200000401:0x194:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826326.836062:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.836070:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.836079:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.836086:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826326.836179:0:225373:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e50:0x0](ffff9057dfdf5b88) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826326.836192:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826326.837148:0:225373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x196:0x0]) in obj [0x240000400:0x196:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826326.837172:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826326.837190:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826326.841914:0:225373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332616832:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826326.842049:0:225373:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef210a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826326.861090:0:225373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332616832:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826326.862533:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826326.862558:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07400/0x95194e876080164e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b138 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826326.862591:0:225373:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x194:0x0]) in obj [0x200000401:0x194:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826326.862610:0:225373:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826326.862627:0:225373:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826326.871247:0:225373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332616960:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826326.871424:0:225373:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef210a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826326.878700:0:225373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332616960:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826326.880546:0:225373:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826326.880581:0:225373:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826326.943458:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346243840 00000100:00100000:3.0:1787826326.943470:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346243840 00000100:00100000:3.0:1787826326.943544:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43601 00000100:00100000:3.0:1787826326.943557:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346243840:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826326.943581:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826326.943695:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826326.943704:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346243840:12345-192.168.203.156@tcp:104: Request processed in 149us (375us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826326.943711:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43601 00010000:00010000:0.0:1787826326.943949:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826326.943960:0:15673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.943984:0:15673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826326.943991:0:15673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944001:0:15673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944011:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944173:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944182:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826326.944264:0:15673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944274:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e8760801655 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x194:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b146 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826326.944300:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.944302:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.944304:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826326.944324:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826326.944348:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332618368:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826326.946371:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332617984:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826326.956338:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332618368:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826326.964710:0:225373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332618752:192.168.203.156@tcp:35:rm.0 00000002:00100000:3.0:1787826326.964754:0:225373:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb238380 x1874663332617472/t4294985420(4294985420) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826342 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826326.964821:0:225373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07000/0x95194e8760801647 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b12a expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826326.964840:0:225373:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07200/0x95194e87608015f3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b004 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826326.964886:0:225373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e8760801663 lrc: 1/0,0 mode: --/CR res: [0x240000403:0x1e50:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b162 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826326.964894:0:225373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07000/0x95194e8760801647 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b12a expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826326.964909:0:225373:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07200/0x95194e87608015f3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b004 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826326.964916:0:225373:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1787826326.964920:0:225373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e8760801663 lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e50:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b162 expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826326.964933:0:225373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07000/0x95194e8760801647 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b12a expref: -99 pid: 225373 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826326.964942:0:225373:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07200/0x95194e87608015f3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b004 expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1787826326.964975:0:225373:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826326.964982:0:225373:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb221180 x1874663332619008/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826326.965010:0:225373:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332619008:192.168.203.156@tcp:36:rm.0 00000100:00100000:3.0:1787826326.965111:0:225373:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00010000:00010000:0.0:1787826326.973195:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826326.973203:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973214:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2ce00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826326.973245:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973257:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973266:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973336:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973344:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826326.973373:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973380:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e876080160f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b043 expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826326.973415:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826326.973419:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973426:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f8c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826326.973429:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973436:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973443:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973471:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973477:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826326.973484:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973491:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8c00/0x95194e87608015e5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4afda expref: -99 pid: 225335 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826326.973514:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.973515:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.973517:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826326.973533:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332619264:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826326.981696:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332619264:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826326.984657:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346246272 00000100:00100000:3.0:1787826326.984671:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346246272 00000100:00100000:3.0:1787826326.984729:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43604 00000100:00100000:3.0:1787826326.984741:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346246272:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826326.984765:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826326.985043:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826326.985052:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346246272:12345-192.168.203.156@tcp:104: Request processed in 312us (469us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826326.985060:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43604 00010000:00010000:0.0:1787826326.985213:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826326.985241:0:6167:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985253:0:6167:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d800 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826326.985259:0:6167:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985278:0:6167:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985288:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985373:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 2/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985392:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826326.985427:0:6167:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985441:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d800/0x95194e8760801616 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e50:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b05f expref: -99 pid: 225337 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826326.985460:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.985463:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826326.985465:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826326.985474:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826326.985492:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332619392:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826326.991214:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346246528 00000100:00100000:0.0:1787826326.991257:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332619392:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826326.991417:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346246528 00000100:00100000:0.0:1787826326.993512:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332619520:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826326.999474:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332619520:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826327.006861:0:225373:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225373:1874663332619008:192.168.203.156@tcp:36:rm.0 00000080:00200000:3.0:1787826327.007150:0:225373:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e50:0x0](ffff9057dfdf5b88) 00000100:00100000:0.0:1787826327.101568:0:225381:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957411 gen 1 00000100:00100000:0.0:1787826327.101605:0:225381:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:225381:1874663332619648:192.168.203.156@tcp:41:stat.0 00800000:00000004:0.0:1787826327.101622:0:225381:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: update ffff9057c6ef0450 cache blocks 1793350/1916558 objects 519503/534195 00000080:00000004:0.0:1787826327.101630:0:225381:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057c7539000: update ffff9057c6ef37c0 cache blocks 1793350/1916558 objects 519503/534195 00000080:00000004:0.0:1787826327.101635:0:225381:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793350/1916558 objects 519503/534195 00000080:00200000:0.0:1787826327.137195:0:225385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826327.137401:0:225385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826327.137466:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.137477:0:225385:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.137492:0:225385:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826327.137516:0:225385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225385:1874663332619776:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826327.137703:0:225385:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef210a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826327.140969:0:225385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225385:1874663332619776:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826327.140994:0:225385:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141006:0:225385:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141017:0:225385:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141027:0:225385:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141039:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141046:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141053:0:225385:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826327.141067:0:225385:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f100 x1874663332619776/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826343 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826327.141081:0:225385:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141092:0:225385:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826327.141098:0:225385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826327.141124:0:225385:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826327.141130:0:225385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801678 00000080:00010000:0.0:1787826327.141135:0:225385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826327.141138:0:225385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801678 from it ffffb512c7a13b58 00010000:00010000:0.0:1787826327.141140:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141148:0:225385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826327.141170:0:225385:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826327.141186:0:225385:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:0.0:1787826327.141190:0:225385:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:0.0:1787826327.141209:0:225385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826327.141286:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141294:0:225385:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.141306:0:225385:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826327.141324:0:225385:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225385:1874663332619904:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826327.141403:0:225385:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef210a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826327.148331:0:225385:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225385:1874663332619904:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826327.148364:0:225385:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148392:0:225385:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148399:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148408:0:225385:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148417:0:225385:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148425:0:225385:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826327.148431:0:225385:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148438:0:225385:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148446:0:225385:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148454:0:225385:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826327.148463:0:225385:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.148470:0:225385:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e876080167f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225385 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826327.148492:0:225385:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057f158f100 x1874663332619904/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826343 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826327.148508:0:225385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826327.148545:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826327.148560:0:225385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826327.148571:0:225385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801678 from it ffffb512c7a13930 00010000:00010000:0.0:1787826327.148575:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826327.148584:0:225385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826327.148754:0:225385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826327.148774:0:225385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826327.148782:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826327.148798:0:225385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826327.148804:0:225385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826327.148809:0:225385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826327.148813:0:225385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801678 00000080:00010000:0.0:1787826327.148818:0:225385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801678 from it ffffb512c7a13be8 00010000:00010000:0.0:1787826327.148822:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826327.148830:0:225385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826327.148839:0:225385:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826327.148848:0:225385:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:0.0:1787826327.148877:0:225385:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826327.148883:0:225385:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826327.148890:0:225385:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826327.148897:0:225385:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826327.149050:0:225385:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826327.149059:0:225385:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826327.149063:0:225385:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826327.149066:0:225385:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801678 00000080:00010000:0.0:1787826327.149070:0:225385:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801678 from it ffffb512c7a13cb8 00010000:00010000:0.0:1787826327.149074:0:225385:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826327.149082:0:225385:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826327.172620:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826327.172668:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826327.173270:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826327.173280:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173291:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3a00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826327.173297:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173308:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173317:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173423:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173432:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826327.173467:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173477:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e8760801678 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b21f expref: -99 pid: 225385 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826327.173510:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826327.173513:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826327.173517:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826327.173529:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826327.173878:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332620416:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826327.183495:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332620416:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826327.264546:0:225374:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), inode mode 41ed mask 1 00000080:00200000:0.0:1787826327.264579:0:225374:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), inode mode 41ed mask 3 00000080:00200000:0.0:1787826327.264585:0:225374:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), file ffff9057c56d6900, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826327.264595:0:225374:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), intent=open|creat 00000002:00010000:0.0:1787826327.264630:0:225374:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x200000403:0x1dbd:0x0]) in obj [0x200000401:0x195:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826327.264644:0:225374:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d600/0x95194e87608016a9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b2ce expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826327.267092:0:225374:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d600/0x95194e87608016a9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b2ce expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826327.267114:0:225374:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826327.267118:0:225374:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d600/0x95194e87608016a9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b2ce expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826327.267533:0:225374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x195:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225374 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.267543:0:225374:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225374 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826327.267558:0:225374:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826327.267574:0:225374:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f100 x1874663332621440/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826327.267583:0:225374:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225374 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826327.267611:0:225374:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225374:1874663332621440:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826327.268547:0:225374:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef210a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826327.274670:0:225374:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225374:1874663332621440:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826327.274704:0:225374:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/0,1 mode: --/CW res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274738:0:225374:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274816:0:225374:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274899:0:225374:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274914:0:225374:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274931:0:225374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274945:0:225374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.274961:0:225374:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826327.275057:0:225374:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158f100 x1874663332621440/t8589957417(8589957417) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826343 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826327.275072:0:225374:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f100 x1874663332621440/t8589957417(8589957417) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826343 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826327.275088:0:225374:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826327.275101:0:225374:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.275114:0:225374:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7613 (0 0) 00000080:00200000:0.0:1787826327.275121:0:225374:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000401:0x195:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826327.275420:0:225374:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), intent=open|creat 00000080:00200000:0.0:1787826327.275465:0:225374:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dbd:0x0](ffff9057dfe042c8) 00000080:00200000:0.0:1787826327.275573:0:225374:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826327.275578:0:225374:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000403:0x1dbd:0x0](ffff9057dfe042c8) 00000080:00010000:0.0:1787826327.275582:0:225374:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbd:0x0](ffff9057dfe042c8) for lock 0x95194e87608016b0 00000080:00010000:0.0:1787826327.275595:0:225374:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbd:0x0](ffff9057dfe042c8) 00000080:00200008:0.0:1787826327.275613:0:225374:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1dbd:0x0](ffff9057dfe042c8), flags 101101 00000002:00100000:0.0:1787826327.275640:0:225374:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f100 x1874663332621440/t8589957417(8589957417) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826343 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826327.275686:0:225374:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016b0 from it ffff9057f9ad4840 00010000:00010000:0.0:1787826327.275693:0:225374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.275704:0:225374:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826327.275732:0:225374:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1dbd:0x0](ffff9057dfe042c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1787826327.276368:0:225374:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn:[0x200000403:0x1dbd:0x0], ppos: 0, count: 6 00000080:00200000:0.0:1787826327.276397:0:225374:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826327.276406:0:225374:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:0.0:1787826327.276413:0:225374:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: write ppos: 0, bytes: 6 00000080:00200000:0.0:1787826327.276419:0:225374:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826327.276431:0:225374:0:(cl_io.c:214:cl_io_rw_init()) header@00000000232e3470[0x110, 1, [0x200000403:0x1dbd:0x0] hash] 00000020:00200000:0.0:1787826327.276434:0:225374:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:0.0:1787826327.276441:0:225374:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826327.276458:0:225374:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787826327.276467:0:225374:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:0.0:1787826327.276471:0:225374:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826327.276492:0:225374:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787826327.276511:0:225374:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826327.276517:0:225374:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d330, osc lock ffff9057c8b83560, flags 0x0 00010000:00010000:0.0:1787826327.276585:0:225374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.276597:0:225374:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.276609:0:225374:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826327.276636:0:225374:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225374:1874663332621568:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826327.276753:0:225374:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f15263c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826327.281997:0:225374:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225374:1874663332621568:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826327.282034:0:225374:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.282059:0:225374:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.282074:0:225374:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.282094:0:225374:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826327.282110:0:225374:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.283097:0:225374:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826327.283116:0:225374:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826327.334013:0:225388:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826327.334064:0:225388:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787826327.334081:0:6634:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826327.334092:0:6634:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c600/0x95194e876080168d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b288 expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826327.334142:0:6634:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826327.334167:0:6634:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbd:0x0]: losing layout lock 00010000:00010000:3.0:1787826327.334199:0:6634:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2cc00/0x95194e8760801694 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b2a4 expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826327.334291:0:6634:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c600/0x95194e876080168d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b288 expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826327.334300:0:6634:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608016b0 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b2ea expref: -99 pid: 225374 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826327.334305:0:6634:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2cc00/0x95194e8760801694 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b2a4 expref: -99 pid: 225374 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826327.334310:0:6634:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:3.0:1787826327.334340:0:6634:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:fc0b407b-a541-420d-8712-b9c8da18fe8c:6634:1874663332622592:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826327.334486:0:6634:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00010000:00010000:3.0:1787826327.335711:0:225390:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826327.335736:0:225390:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787826327.336416:0:225387:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00000100:00100000:0.0:1787826329.006588:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332622976:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006721:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006747:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006764:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006782:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332624000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006813:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:0.0:1787826329.006819:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623104:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006843:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006863:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.006879:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.022966:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623104:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.022989:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023004:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023015:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023026:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332622976:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023039:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023051:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332623488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023063:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332623744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826329.023074:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332624000:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826329.378669:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826330.020785:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826330.020804:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826330.020836:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826330.020850:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826330.020854:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787826330.020891:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000a4a3ed8b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826330.020916:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000a4a3ed8b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826330.020993:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1787826330.021099:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158c380 - object 0x2c0000401:1313 offset 0<>6 00000100:00100000:0.0:1787826330.021178:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332624128:192.168.203.156@tcp:4:bash.0 00000100:00100000:0.0:1787826330.021332:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826330.021339:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332624256:192.168.203.156@tcp:16:kworker.0 00000100:00100000:0.0:1787826330.032833:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332624256:192.168.203.156@tcp:16:kworker.0 00000080:00200000:3.0:1787826330.033103:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826330.033139:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826330.033152:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826330.033184:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000008feb2e5f ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826330.033206:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000008feb2e5f ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826330.033322:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826330.033429:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb220700 - object 0x280000400:1230 offset 0<>6 00000100:00100000:3.0:1787826330.033502:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332624512:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826330.033600:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826330.033607:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332624384:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826330.037762:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332624512:192.168.203.156@tcp:16:kworker.0 00000080:00200000:0.0:1787826330.037880:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1787826330.053125:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000008feb2e5f ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826330.053182:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000008feb2e5f ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826330.053565:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332624384:192.168.203.156@tcp:4:bash.0 00000008:00010000:0.0:1787826330.056021:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000a4a3ed8b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787826330.056071:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000a4a3ed8b ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e87608016b7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b2f8 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787826330.056111:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332624128:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826330.308996:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826330.323447:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332624640:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826330.533526:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1db8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826330.533559:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826330.533596:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1db8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826331.275799:0:225555:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826331.275841:0:225555:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826331.275923:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826331.275936:0:225555:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826331.275952:0:225555:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826331.275988:0:225555:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332624768:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826331.276129:0:225555:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.311375:0:225555:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332625280:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826331.311520:0:225555:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f15263c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.317213:0:225555:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332625280:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826331.317369:0:225555:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317385:0:225555:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317400:0:225555:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317413:0:225555:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317427:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317436:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317444:0:225555:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826331.317458:0:225555:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de3100 x1874663332625280/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826331.317473:0:225555:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317489:0:225555:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/405 (0 0) 00010000:00010000:0.0:1787826331.317498:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.317507:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e87608016e8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b37d expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.318010:0:225555:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e52:0x0], updating i_size 8192 00000080:00010000:0.0:1787826331.318029:0:225555:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) for remote lock 0x95194e87608016da 00000080:00010000:0.0:1787826331.318036:0:225555:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) for lock 0x95194e87608016d3 00000080:00010000:0.0:1787826331.318049:0:225555:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) 00000080:00010000:0.0:1787826331.318053:0:225555:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016d3 from it ffffb512c7af3ac8 00010000:00010000:0.0:1787826331.318057:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608016d3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b353 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.318067:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06800/0x95194e87608016d3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b353 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826331.318077:0:225555:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e87608016da from it ffffb512c7af3ac8 00010000:00010000:0.0:1787826331.318080:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e87608016da lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b361 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.318089:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e87608016da lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b361 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.318310:0:225555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), inode mode 41ed mask 1 00000080:00200000:0.0:1787826331.318342:0:225555:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), file ffff9057c8ce9800, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1787826331.318359:0:225555:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), intent=open 00000002:00010000:0.0:1787826331.318388:0:225555:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x195:0x0], intent: open flags 0100001 00010000:00010000:0.0:1787826331.318458:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608016ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.318467:0:225555:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608016ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826331.318481:0:225555:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826331.318491:0:225555:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332625536/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826331.318502:0:225555:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608016ef lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.318530:0:225555:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332625536:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826331.320264:0:225555:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f15263c0 going to sleep for 11 seconds 00010000:00010000:0.0:1787826331.426557:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826331.796679:0:225557:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826331.796738:0:225557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826331.796823:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.796836:0:225557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.796849:0:225557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.796882:0:225557:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625664:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826331.797028:0:225557:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.800690:0:225557:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625664:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826331.800779:0:225557:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800795:0:225557:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800810:0:225557:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800892:0:225557:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800907:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800915:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800924:0:225557:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826331.800939:0:225557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158e300 x1874663332625664/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826331.800953:0:225557:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.800968:0:225557:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826331.800977:0:225557:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826331.801013:0:225557:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826331.801021:0:225557:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608016f6 00000080:00010000:0.0:1787826331.801028:0:225557:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826331.801031:0:225557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016f6 from it ffffb512c7bebbd0 00010000:00010000:0.0:1787826331.801035:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.801050:0:225557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.801095:0:225557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826331.801121:0:225557:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=24 00000080:00200000:0.0:1787826331.801126:0:225557:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:0.0:1787826331.801153:0:225557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826331.801203:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.801211:0:225557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.801492:0:225557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.801530:0:225557:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625792:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826331.801635:0:225557:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.804878:0:225557:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625792:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826331.804915:0:225557:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.804960:0:225557:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.804971:0:225557:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.804985:0:225557:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.804998:0:225557:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805013:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805022:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805029:0:225557:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826331.805042:0:225557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158e300 x1874663332625792/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826331.805054:0:225557:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805067:0:225557:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7762 (0 0) 00000002:00010000:0.0:1787826331.805086:0:225557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e52:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826331.805140:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805148:0:225557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.805158:0:225557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.805178:0:225557:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625920:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826331.805511:0:225557:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.819512:0:225557:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057c7539000: committing for last_committed 4294985426 gen 1 00000100:00100000:0.0:1787826331.819569:0:225557:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332625920:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826331.819596:0:225557:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819611:0:225557:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819637:0:225557:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819651:0:225557:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819674:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819684:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819692:0:225557:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826331.819706:0:225557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158d180 x1874663332625920/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826331.819721:0:225557:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819735:0:225557:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7762 (0 0) 00000080:00200000:0.0:1787826331.819754:0:225557:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826331.819802:0:225557:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e52:0x0], updating i_size 4096 00000080:00200000:0.0:1787826331.819808:0:225557:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e52:0x0](ffff9057dfdd7448): rc = 0 00000080:00010000:0.0:1787826331.819818:0:225557:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd7448) for remote lock 0x95194e8760801704 00000080:00010000:0.0:1787826331.819822:0:225557:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd7448) for lock 0x95194e87608016fd 00000080:00010000:0.0:1787826331.819836:0:225557:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd7448) 00000080:00010000:0.0:1787826331.819840:0:225557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016fd from it ffffb512c7bebaa0 00010000:00010000:0.0:1787826331.819845:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.819855:0:225557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e87608016fd lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3bc expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826331.822187:0:225557:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801704 from it ffffb512c7bebaa0 00010000:00010000:0.0:1787826331.822206:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.822236:0:225557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.822298:0:225557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), inode mode 41ed mask 1 00000080:00200000:0.0:1787826331.822333:0:225557:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826331.822378:0:225557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826331.822396:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.822411:0:225557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.822419:0:225557:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826331.822426:0:225557:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826331.822430:0:225557:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608016f6 00000080:00010000:0.0:1787826331.822435:0:225557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016f6 from it ffffb512c7bebbd0 00010000:00010000:0.0:1787826331.822438:0:225557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.822445:0:225557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826331.822453:0:225557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826331.822460:0:225557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=24 00000080:00200000:0.0:1787826331.822466:0:225557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), inode mode 41ed mask 1 00000080:00200000:0.0:1787826331.822484:0:225557:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), flags=0 00000080:00200000:0.0:1787826331.822488:0:225557:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd7448), intent=getattr 00000002:00010000:0.0:1787826331.822504:0:225557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x0:0x0:0x0]) in obj [0x240000400:0x197:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826331.822565:0:225557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e876080170b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x197:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.822573:0:225557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e876080170b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.822585:0:225557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e876080170b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.824030:0:225557:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332626048:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826331.824189:0:225557:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.831190:0:225557:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225557:1874663332626048:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826331.831376:0:225557:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e876080170b lrc: 4/1,0 mode: --/CR res: [0x240000400:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b3e6 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.849984:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332626304:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826331.866278:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826331.866335:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346255360 00000100:00100000:3.0:1787826331.866358:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346255360 00000100:00100000:3.0:1787826331.866428:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43608 00000100:00100000:3.0:1787826331.866441:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346255360:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826331.866465:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866559:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826331.866567:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346255360:12345-192.168.203.156@tcp:104: Request processed in 129us (304us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826331.866574:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43608 00010000:00010000:3.0:1787826331.866605:0:49150:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826331.866612:0:49150:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866621:0:49150:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b07c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826331.866627:0:49150:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866636:0:49150:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866645:0:49150:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866719:0:49150:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866726:0:49150:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826331.866750:0:49150:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866759:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07c00/0x95194e87608016e1 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b36f expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826331.866798:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826331.866800:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826331.866802:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826331.866820:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332626688:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826331.867095:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332626304:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826331.871975:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332626816:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826331.875125:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826331.875281:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332626688:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826331.877465:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000080:00200000:0.0:1787826331.879979:0:225555:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbd:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826331.879986:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_release():245 00000020:00010000:0.0:1787826331.879990:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826331.879996:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057edd1eeb0 (ffff9057c00253f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826331.880001:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826331.880003:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057edd1eeb0 (ffff9057ca858b48/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826331.880018:0:225555:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbd:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826331.880158:0:225555:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbd:0x0], ppos: 0, count: 4194304 00000080:00200000:0.0:1787826331.880185:0:225555:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:0.0:1787826331.880205:0:225555:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:0.0:1787826331.880211:0:225555:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826331.880276:0:225555:0:(cl_io.c:214:cl_io_rw_init()) header@00000000dc94c6a4[0x110, 1, [0x200000403:0x1dbd:0x0] hash] 00000020:00200000:0.0:1787826331.880278:0:225555:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826331.880283:0:225555:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826331.880300:0:225555:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826331.880304:0:225555:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826331.880321:0:225555:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826331.880336:0:225555:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826331.880342:0:225555:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d830, osc lock ffff9057c8b82980, flags 0x0 00010000:00010000:0.0:1787826331.880418:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.880430:0:225555:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.880440:0:225555:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.880466:0:225555:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332626944:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826331.880560:0:225555:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:3.0:1787826331.883386:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826331.883508:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627072:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826331.884973:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826331.884991:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332627200:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826331.886615:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627072:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826331.891077:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826331.891126:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826331.891170:0:6159:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826331.891188:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346256896 00000100:00100000:3.0:1787826331.891191:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346256896 00000100:00100000:0.0:1787826331.891304:0:225555:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332626944:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826331.891332:0:225555:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.891360:0:225555:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.891373:0:225555:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826331.893246:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43613 00000100:00100000:3.0:1787826331.893271:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d86dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346256896:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826331.893292:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608016ef lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4b410 expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826331.893418:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826331.893426:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d86dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346256896:12345-192.168.203.156@tcp:104: Request processed in 167us (2298us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826331.893433:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43613 00000100:00100000:0.0:1787826331.894479:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627328:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826331.899585:0:225555:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.899605:0:225555:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.899627:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826331.899643:0:225555:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.899658:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.899667:0:225555:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826331.899707:0:225555:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000020:00010000:0.0:1787826331.899724:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057caa28e48 (ffff9057ca858b48/0) at cl_lock_release():245 00000020:00010000:0.0:1787826331.899728:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057caa28e48 (ffff9057ca858b48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826331.899732:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d830 (ffff9057c00253f8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826331.899738:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.899749:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826331.899778:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057caa28e48 (ffff9057ca858b48/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826331.899782:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d830 (ffff9057ca858b48/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826331.899791:0:225555:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826331.899805:0:225555:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826331.899809:0:225555:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 1 00000080:00200000:0.0:1787826331.899813:0:225555:0:(file.c:2245:ll_file_io_generic()) f45j.sanityn: restart read from ppos=0 bytes=4194304 retries=999 ret=0: rc = 0 00000080:00200000:0.0:1787826331.899820:0:225555:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826331.899827:0:225555:0:(cl_io.c:214:cl_io_rw_init()) header@00000000dc94c6a4[0x110, 1, [0x200000403:0x1dbd:0x0] hash] 00000020:00200000:0.0:1787826331.899830:0:225555:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826331.899835:0:225555:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbd:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826331.899975:0:225555:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000403:0x1dbd:0x0](ffff9057dfdd00c8) 00000002:00010000:0.0:1787826331.899987:0:225555:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dbd:0x0]) in obj [0x200000403:0x1dbd:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826331.900042:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.900051:0:225555:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.900064:0:225555:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225555 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826331.900087:0:225555:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332627456:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826331.900271:0:225555:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826331.910256:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627328:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826331.910927:0:225555:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332627456:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826331.911154:0:225555:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.911173:0:225555:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826331.911191:0:225555:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158e300 x1874663332627456/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826331.911203:0:225555:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826331.911217:0:225555:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.911267:0:225555:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 16 res 8589935619/7613 (0 0) 00000080:00200000:0.0:1787826331.911275:0:225555:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000403:0x1dbd:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826331.911293:0:225555:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbd:0x0](ffff9057dfdd00c8) for lock 0x95194e8760801727 00000080:00010000:0.0:1787826331.911302:0:225555:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801727 from it ffffb512c7af3af8 00010000:00010000:0.0:1787826331.911308:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826331.911318:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911360:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911373:0:225555:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826331.911383:0:225555:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000403:0x1dbd:0x0](000000002801ae2e) being reconfigured ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.911436:0:225555:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbd:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826331.911442:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911450:0:225555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826331.911477:0:225555:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826331.911482:0:225555:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826331.911505:0:225555:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826331.911522:0:225555:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826331.911527:0:225555:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d230, osc lock ffff9057c8b82980, flags 0x0 00010000:00010000:0.0:1787826331.911548:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911565:0:225555:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911579:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911588:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911595:0:225555:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826331.911606:0:225555:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826331.911702:0:225555:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbd:0x0] 00000008:00010000:0.0:1787826331.911709:0:225555:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826331.911712:0:225555:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5deb0, osc lock ffff9057c8b82390, flags 0x400001002 00010000:00010000:0.0:1787826331.911721:0:225555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911730:0:225555:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826331.911739:0:225555:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826331.911743:0:225555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826331.911755:0:225555:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826331.911762:0:225555:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826331.911769:0:225555:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbd:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826331.911774:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_release():245 00000020:00010000:0.0:1787826331.911778:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826331.911781:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5deb0 (ffff9057c00253f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826331.911787:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057e0209280 (ffff9057ca858b48/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826331.911790:0:225555:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5deb0 (ffff9057ca858b48/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826331.911801:0:225555:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1787826331.911863:0:225555:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff9057c644b3c0 00000008:00010000:0.0:1787826331.911908:0:215335:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9057c644b3c0 00010000:00010000:0.0:1787826331.911934:0:215335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:0.0:1787826331.912187:0:215335:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158e300 - object 0x2c0000401:1313 offset 0<>4096 00000100:00100000:0.0:1787826331.912283:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332627584:192.168.203.156@tcp:3:cat.0 00000008:00010000:0.0:1787826331.915743:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9057c644b3c0 00000020:00200000:0.0:1787826331.915764:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1787826331.915819:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9057c644b3c0 00000100:00100000:0.0:1787826331.915852:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332627584:192.168.203.156@tcp:3:cat.0 00000100:00100000:0.0:1787826331.922086:0:225555:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225555:1874663332627840:192.168.203.156@tcp:35:cat.0 00000002:00100000:0.0:1787826331.922143:0:225555:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158ed80 x1874663332625536/t8589957423(8589957423) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826347 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826332.421105:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826332.423270:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346259072 00000100:00100000:3.0:1787826332.423292:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346259072 00000100:00100000:3.0:1787826332.423398:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43614 00000100:00100000:3.0:1787826332.423410:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb222300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346259072:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826332.423433:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000033444554/0x95194e8760801720 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4b441 expref: -99 pid: 225555 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826332.423545:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826332.423552:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb222300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346259072:12345-192.168.203.156@tcp:104: Request processed in 145us (2465us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826332.423557:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43614 00000100:00100000:0.0:1787826332.425043:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627968:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826332.435115:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332627968:192.168.203.156@tcp:103:ldlm_bl.0 00000002:00010000:0.0:1787826333.347802:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x195:0x0]) in obj [0x200000401:0x195:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826333.347835:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826333.347925:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826333.350537:0:225593:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332628864:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826333.352295:0:225593:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.378186:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332628864:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826333.379072:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826333.379093:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826333.379326:0:225593:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), flags 704000 00000002:00100000:0.0:1787826333.379430:0:225593:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb23bb80 x1874663332628608/t4294985429(4294985429) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826349 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826333.379443:0:225593:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826333.379449:0:225593:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332629120/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826333.379480:0:225593:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629120:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826333.382319:0:225593:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.397571:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629120:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826333.397632:0:225593:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb23bb80 x1874663332628608/t4294985429(4294985429) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826333.397734:0:225593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826333.397759:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.397782:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.397798:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826333.397807:0:225593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826333.397814:0:225593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826333.397818:0:225593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608016cc 00000080:00010000:0.0:1787826333.397827:0:225593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016cc from it ffffb512c7c2bbf8 00010000:00010000:0.0:1787826333.397832:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.397842:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826333.397869:0:225593:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826333.397878:0:225593:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826333.397888:0:225593:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), inode mode 41ed mask 44 00000080:00200008:0.0:1787826333.397898:0:225593:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), flags 704400 00000002:00010000:0.0:1787826333.397949:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e52:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826333.398018:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.398026:0:225593:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826333.398038:0:225593:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826333.398045:0:225593:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332629376/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826333.398056:0:225593:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.398079:0:225593:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629376:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826333.398730:0:225593:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.416198:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629376:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826333.416287:0:225593:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.416303:0:225593:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.416319:0:225593:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.416332:0:225593:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.416361:0:225593:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158ed80 x1874663332629376/t4294985431(4294985431) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826349 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826333.416376:0:225593:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ed80 x1874663332629376/t4294985431(4294985431) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826349 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.416392:0:225593:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.416404:0:225593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7762 (0 0) 00000080:00200000:0.0:1787826333.416412:0:225593:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826333.416464:0:225593:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e52:0x0], updating i_size 4096 00000080:00010000:0.0:1787826333.416472:0:225593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) for lock 0x95194e876080174a 00000080:00010000:0.0:1787826333.416480:0:225593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080174a from it ffffb512c7c2bbd0 00010000:00010000:0.0:1787826333.416485:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.416498:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826333.416533:0:225593:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ed80 x1874663332629376/t4294985431(4294985431) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826349 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826333.416650:0:225593:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826333.416660:0:225593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08),name=d45j.sanityn 00000002:00010000:0.0:1787826333.416686:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e52:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.416699:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e876080172e lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b487 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.416714:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e876080172e lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b487 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826333.416724:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x197:0x0]) in obj [0x240000400:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.416731:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416739:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416748:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416757:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826333.416767:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x195:0x0]) in obj [0x200000401:0x195:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.416776:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416785:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416793:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826333.416800:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826333.416811:0:225593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826333.416817:0:225593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) 00000080:00010000:0.0:1787826333.416821:0:225593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e52:0x0](ffff9057dfdd4b08) for lock 0x95194e876080172e 00000080:00010000:0.0:1787826333.416825:0:225593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080172e from it ffffb512c7c2bcd8 00010000:00010000:0.0:1787826333.416828:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e876080172e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b487 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.416836:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e876080172e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b487 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826333.416848:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x197:0x0]) in obj [0x240000400:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.416868:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416876:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416883:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416891:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826333.416899:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x195:0x0]) in obj [0x200000401:0x195:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.416903:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416915:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416923:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826333.416930:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000100:00100000:3.0:1787826333.426009:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629504:192.168.203.156@tcp:41:rm.0 00800000:00000004:3.0:1787826333.426052:0:225593:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793350/1916558 objects 519505/534200 00000080:00000004:3.0:1787826333.426061:0:225593:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793350/1916558 objects 519505/534200 00000080:00000004:3.0:1787826333.426067:0:225593:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793350/1916558 objects 519505/534200 00000080:00200000:3.0:1787826333.426387:0:225593:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826333.426428:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:3.0:1787826333.426485:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x197:0x0]) in obj [0x240000400:0x197:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826333.426498:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.426509:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:3.0:1787826333.426524:0:225593:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:3.0:1787826333.426530:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.426540:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:3.0:1787826333.426559:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x195:0x0]) in obj [0x200000401:0x195:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826333.426567:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.426576:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:3.0:1787826333.426589:0:225593:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:3.0:1787826333.426592:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.426600:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080173c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x195:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4a3 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1787826333.427759:0:225593:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:3.0:1787826333.427795:0:225593:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), inode mode 41ed mask 1 00000080:00200000:3.0:1787826333.427818:0:225593:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), flags=0 00000080:00200000:3.0:1787826333.427823:0:225593:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), intent=getattr 00000002:00010000:3.0:1787826333.427849:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x195:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826333.428096:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.428171:0:225593:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.428183:0:225593:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225593 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826333.428213:0:225593:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629632:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826333.429171:0:225593:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826333.435147:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332629632:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826333.435187:0:225593:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x195:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435216:0:225593:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435450:0:225593:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435465:0:225593:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435478:0:225593:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435494:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435503:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435511:0:225593:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826333.435525:0:225593:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222a00 x1874663332629632/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826333.435538:0:225593:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8200/0x95194e8760801751 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4e2 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.435550:0:225593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7763 (0 0) 00010000:00010000:3.0:1787826333.442464:0:225593:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801758 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b4f0 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.442511:0:225593:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801758 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b4f0 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.442518:0:225593:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826333.442520:0:225593:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06e00/0x95194e8760801758 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e53:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b4f0 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1787826333.442548:0:225593:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826333.442558:0:225593:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb223b80 x1874663332630144/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826333.442577:0:225593:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332630144:192.168.203.156@tcp:36:rm.0 00000100:00100000:3.0:1787826333.442623:0:225593:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826333.447837:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826333.478556:0:225593:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225593:1874663332630144:192.168.203.156@tcp:36:rm.0 00000080:00200000:3.0:1787826333.478633:0:225593:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826333.478649:0:225593:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826333.478685:0:225593:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826333.478694:0:225593:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e53:0x0](ffff9057dfdd5b88) 00000080:00200008:3.0:1787826333.478806:0:225593:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e52:0x0](ffff9057dfdd4b08), flags 704000 00010000:00010000:3.0:1787826333.478843:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e876080174a lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e52:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b4bf expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826333.479096:0:225593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826333.479187:0:225593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826333.479201:0:225593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.479211:0:225593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1787826333.479475:0:225593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826333.479485:0:225593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826333.479489:0:225593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608016cc 00000080:00010000:3.0:1787826333.479498:0:225593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608016cc from it ffffb512c7c2bc10 00010000:00010000:3.0:1787826333.479502:0:225593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1787826333.479511:0:225593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1787826333.479528:0:225593:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000100:00100000:3.0:1787826333.479927:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332630400:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826333.485443:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332630400:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826333.489406:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826333.489447:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826333.489489:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346261376 00000100:00100000:3.0:1787826333.489501:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346261376 00000100:00100000:3.0:1787826333.489553:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346261504 00000100:00100000:3.0:1787826333.489556:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346261504 00000100:00100000:3.0:1787826333.489574:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43616 00000100:00100000:3.0:1787826333.489584:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346261376:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826333.489616:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826333.489711:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826333.489719:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6532:x1874663346261376:12345-192.168.203.156@tcp:104: Request processed in 136us (328us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826333.489726:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43616 00000100:00100000:3.0:1787826333.489747:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43617 00000100:00100000:3.0:1787826333.489753:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb220e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346261504:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826333.489763:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826333.489792:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826333.489797:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb220e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346261504:12345-192.168.203.156@tcp:104: Request processed in 45us (350us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826333.489804:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43617 00010000:00010000:0.0:1787826333.490036:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826333.490045:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490054:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826333.490060:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490068:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490077:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490148:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490157:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826333.490187:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490194:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e87608016f6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b3a0 expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.490217:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826333.490263:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490270:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826333.490273:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490280:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490288:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490317:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490324:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826333.490330:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490349:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e87608016cc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b337 expref: -99 pid: 225555 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826333.490372:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.490374:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.490376:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826333.490396:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332631040:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.490472:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826333.490479:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332631168:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826333.490547:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826333.496841:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332631040:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826333.500044:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826333.501467:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332631168:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826333.506174:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826333.512703:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826333.512712:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512723:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2cc00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826333.512729:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512739:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512753:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512900:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512909:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826333.512936:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512943:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e8760801735 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b495 expref: -99 pid: 225593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826333.512977:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.512979:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.512981:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826333.512999:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332631424:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.516582:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332631424:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826333.661864:0:225605:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826333.661895:0:225605:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.661958:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.661969:0:225605:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.661979:0:225605:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.662003:0:225605:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225605:1874663332631680:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826333.662308:0:225605:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.665154:0:225605:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225605:1874663332631680:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826333.665180:0:225605:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665193:0:225605:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665205:0:225605:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665216:0:225605:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665392:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665401:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665408:0:225605:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.665421:0:225605:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158e680 x1874663332631680/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.665436:0:225605:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665450:0:225605:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826333.665458:0:225605:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826333.665488:0:225605:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826333.665494:0:225605:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080175f 00000080:00010000:0.0:1787826333.665499:0:225605:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826333.665503:0:225605:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080175f from it ffffb512c7c63b58 00010000:00010000:0.0:1787826333.665506:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665515:0:225605:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826333.665542:0:225605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826333.665560:0:225605:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:0.0:1787826333.665565:0:225605:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:0.0:1787826333.665585:0:225605:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826333.665624:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665632:0:225605:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.665641:0:225605:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.665660:0:225605:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225605:1874663332631808:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826333.665762:0:225605:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.680642:0:225605:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225605:1874663332631808:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826333.680664:0:225605:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680683:0:225605:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680690:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680697:0:225605:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680706:0:225605:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680712:0:225605:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826333.680716:0:225605:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680723:0:225605:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680729:0:225605:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680737:0:225605:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826333.680743:0:225605:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.680749:0:225605:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801766 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225605 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.680769:0:225605:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057f158e680 x1874663332631808/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826333.680792:0:225605:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826333.680822:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.680835:0:225605:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826333.680844:0:225605:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080175f from it ffffb512c7c63930 00010000:00010000:0.0:1787826333.680848:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.680868:0:225605:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826333.680953:0:225605:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826333.680967:0:225605:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.680975:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.680984:0:225605:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826333.680990:0:225605:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826333.680995:0:225605:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826333.680998:0:225605:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080175f 00000080:00010000:0.0:1787826333.681004:0:225605:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080175f from it ffffb512c7c63be8 00010000:00010000:0.0:1787826333.681007:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.681015:0:225605:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826333.681024:0:225605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826333.681031:0:225605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:0.0:1787826333.681053:0:225605:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826333.681059:0:225605:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826333.681065:0:225605:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.681072:0:225605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826333.681079:0:225605:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826333.681086:0:225605:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826333.681089:0:225605:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826333.681092:0:225605:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080175f 00000080:00010000:0.0:1787826333.681096:0:225605:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080175f from it ffffb512c7c63cb8 00010000:00010000:0.0:1787826333.681098:0:225605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826333.681106:0:225605:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00100000:3.0:1787826333.694848:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826333.695559:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332632320:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826333.701801:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346264320 00000100:00100000:3.0:1787826333.701810:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346264320 00000100:00100000:3.0:1787826333.701900:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43621 00000100:00100000:3.0:1787826333.701909:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346264320:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826333.701926:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e876080175f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b57c expref: -99 pid: 225605 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826333.701993:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826333.701999:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346264320:12345-192.168.203.156@tcp:104: Request processed in 91us (357us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826333.702007:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43621 00000100:00100000:0.0:1787826333.702323:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332632448:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.704471:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332632320:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.712073:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332632448:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1787826333.758438:0:225594:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826333.758481:0:225594:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826333.758566:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.758580:0:225594:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826333.758594:0:225594:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826333.758631:0:225594:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332632576:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826333.758776:0:225594:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00010000:00010000:0.0:1787826333.785852:0:225594:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.785863:0:225594:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.785873:0:225594:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.785884:0:225594:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786028:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786035:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786043:0:225594:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.786055:0:225594:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb23b800 x1874663332632704/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.786735:0:225594:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786750:0:225594:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7764 (0 0) 00000002:00010000:0.0:1787826333.786768:0:225594:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e54:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826333.786815:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786822:0:225594:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.786830:0:225594:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.786851:0:225594:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332632832:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826333.786980:0:225594:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.791055:0:225594:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332632832:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826333.792367:0:225594:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792381:0:225594:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792393:0:225594:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792404:0:225594:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792418:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792427:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792434:0:225594:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.792448:0:225594:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158e680 x1874663332632832/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.792463:0:225594:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792474:0:225594:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7764 (0 0) 00000080:00200000:0.0:1787826333.792493:0:225594:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1787826333.792555:0:225594:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.792601:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792609:0:225594:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.792617:0:225594:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.792636:0:225594:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332632960:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826333.792732:0:225594:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.804601:0:225594:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332632960:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826333.804630:0:225594:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804641:0:225594:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804652:0:225594:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804662:0:225594:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804676:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804683:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804691:0:225594:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.804702:0:225594:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158e300 x1874663332632960/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.804717:0:225594:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804729:0:225594:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/408 (0 0) 00010000:00010000:0.0:1787826333.804737:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804746:0:225594:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801789 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b61d expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826333.804777:0:225594:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826333.804825:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804833:0:225594:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.804842:0:225594:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225594 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826333.804861:0:225594:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332633088:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826333.804961:0:225594:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.812725:0:225594:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225594:1874663332633088:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826333.812751:0:225594:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812764:0:225594:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812776:0:225594:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812786:0:225594:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812801:0:225594:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812809:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812817:0:225594:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826333.812830:0:225594:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158e300 x1874663332633088/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826349 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826333.812845:0:225594:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812858:0:225594:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/406 (0 0) 00010000:00010000:0.0:1787826333.812879:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.812889:0:225594:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3000/0x95194e8760801790 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b62b expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826333.812923:0:225594:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e54:0x0], updating i_size 8192 00000080:00200000:0.0:1787826333.812928:0:225594:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e54:0x0](ffff9057dfe0ec08): rc = 0 00000080:00010000:0.0:1787826333.812936:0:225594:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) for remote lock 0x95194e8760801782 00000080:00010000:0.0:1787826333.813058:0:225594:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) for lock 0x95194e876080177b 00000080:00010000:0.0:1787826333.813138:0:225594:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) 00000080:00010000:0.0:1787826333.813143:0:225594:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080177b from it ffffb512c7c1bac8 00010000:00010000:0.0:1787826333.813146:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.813155:0:225594:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826333.813164:0:225594:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801782 from it ffffb512c7c1bac8 00010000:00010000:0.0:1787826333.813168:0:225594:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826333.813345:0:225594:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.898714:0:225608:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826333.898752:0:225608:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787826333.898767:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826333.898777:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b5e5 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826333.898821:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801797 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b647 expref: -99 pid: 225594 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826333.898839:0:34880:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbe:0x0]: losing layout lock 00010000:00010000:3.0:1787826333.898864:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.898905:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b5e5 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826333.898915:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801797 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b647 expref: -99 pid: 225594 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.898921:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.898928:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787826333.898949:0:225609:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00000100:00100000:3.0:1787826333.898951:0:34880:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_07:fc0b407b-a541-420d-8712-b9c8da18fe8c:34880:1874663332634496:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826333.899048:0:34880:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00010000:00010000:0.0:1787826333.899102:0:6166:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826333.899108:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2da00/0x95194e87608017a5 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b671 expref: -99 pid: 225594 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826333.899124:0:6166:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e55:0x0]: losing layout lock 00010000:00010000:0.0:1787826333.899144:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.899155:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.899175:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2da00/0x95194e87608017a5 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b671 expref: -99 pid: 225594 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.899188:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801782 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b60f expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.899195:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801704 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e52:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b3ca expref: -99 pid: 225557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.899201:0:6166:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1787826333.899212:0:6166:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:6166:1874663332634624:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.899470:0:6166:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4600 going to sleep for 11 seconds 00010000:00010000:0.0:1787826333.901320:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826333.901342:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826333.901378:0:49149:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbd:0x0]: losing layout lock 00010000:00010000:0.0:1787826333.901420:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 1/0,0 mode: --/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.901429:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826333.901447:0:49149:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_09:398e0287-d3c7-44dd-b1d9-6b85f698e615:49149:1874663332634752:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826333.901521:0:49149:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4780 going to sleep for 11 seconds 00000100:00100000:0.0:1787826333.917074:0:49149:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_09:398e0287-d3c7-44dd-b1d9-6b85f698e615:49149:1874663332634752:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826333.917106:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801727 lrc: 0/0,0 mode: --/CR res: [0x200000403:0x1dbd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b46b expref: -99 pid: 225555 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826333.917148:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.917150:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.917153:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.917916:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.917919:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.917921:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826333.918279:0:34880:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_07:fc0b407b-a541-420d-8712-b9c8da18fe8c:34880:1874663332634496:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826333.918314:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9c00/0x95194e8760801774 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b5e5 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826333.918348:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801797 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b647 expref: -99 pid: 225594 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826333.918362:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d000/0x95194e876080177b lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b601 expref: -99 pid: 225594 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826333.918419:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.918421:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826333.918423:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826334.114450:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332635008:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826334.114542:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1787826334.114549:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332634880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826334.114576:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332635136:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826334.116684:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332634880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826334.120560:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332635008:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826334.125876:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332635136:192.168.203.156@tcp:400:kworker.0 00000080:00200000:0.0:1787826335.140017:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826335.140040:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826335.140073:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826335.140089:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826335.140094:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826335.145294:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826335.145335:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826335.145347:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787826335.145377:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000005831c68 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826335.145408:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000005831c68 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826335.145504:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1787826335.146759:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158e300 - object 0x280000401:1313 offset 0<>6 00000080:00200000:3.0:1787826335.182192:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826335.185299:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e55:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826335.185320:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826335.185368:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000034505496 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826335.185392:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000034505496 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826335.185459:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826335.185557:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb220e00 - object 0x2c0000400:1228 offset 0<>6 00000100:00100000:3.0:1787826335.197473:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332635648:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826335.197530:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000034505496 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826335.197570:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000034505496 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826335.197600:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332635520:192.168.203.156@tcp:4:bash.0 00010000:00010000:0.0:1787826335.523693:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826335.932736:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826335.934279:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826335.934290:0:15672:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826335.934306:0:15672:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c000 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826335.934313:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400010000 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826335.934351:0:15672:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826335.934430:0:15672:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826335.934494:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826335.934502:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826335.934538:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826335.934546:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057c7539000 lock: 0000000032b1f6cf/0x95194e87608016c5 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4b322 expref: -99 pid: 225374 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826335.934572:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826335.934574:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826335.934576:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826336.547428:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826337.570437:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826337.690359:0:225775:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826337.690390:0:225775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826337.690452:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826337.690463:0:225775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826337.690473:0:225775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826337.690495:0:225775:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332635904:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826337.690585:0:225775:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000080:00200000:0.0:1787826337.743962:0:225775:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348) 00000002:00010000:0.0:1787826337.743987:0:225775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826337.744032:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.744040:0:225775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.744049:0:225775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826337.744068:0:225775:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332636288:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826337.744285:0:225775:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826337.754030:0:225775:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332636288:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826337.754057:0:225775:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754069:0:225775:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754081:0:225775:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754091:0:225775:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754185:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754194:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754483:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826337.754503:0:225775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ea00 x1874663332636288/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826353 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826337.754518:0:225775:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754530:0:225775:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/408 (0 0) 00010000:00010000:0.0:1787826337.754537:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826337.754546:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608017c8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6cc expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.192927:0:225777:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826338.192959:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826338.193020:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.193031:0:225777:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.193041:0:225777:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826338.193064:0:225777:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332636800:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826338.194820:0:225777:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826338.200535:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332636800:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826338.200566:0:225777:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200588:0:225777:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200607:0:225777:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200619:0:225777:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200632:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200640:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200647:0:225777:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826338.200658:0:225777:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f480 x1874663332636800/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826338.200671:0:225777:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200683:0:225777:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826338.200690:0:225777:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826338.200717:0:225777:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826338.200723:0:225777:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608017dd 00000080:00010000:0.0:1787826338.200728:0:225777:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826338.200731:0:225777:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017dd from it ffffb512c7e3bbd0 00010000:00010000:0.0:1787826338.200734:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200742:0:225777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.200783:0:225777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826338.200802:0:225777:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=24 00000080:00200000:0.0:1787826338.200807:0:225777:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:0.0:1787826338.200836:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826338.200875:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200882:0:225777:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.200892:0:225777:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826338.200910:0:225777:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332636928:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826338.200987:0:225777:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826338.215632:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332636928:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826338.215661:0:225777:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215688:0:225777:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215695:0:225777:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215707:0:225777:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215717:0:225777:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215730:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215749:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215756:0:225777:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826338.215769:0:225777:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158f480 x1874663332636928/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826338.215783:0:225777:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215798:0:225777:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7764 (0 0) 00000002:00010000:0.0:1787826338.215814:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e54:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826338.215867:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215874:0:225777:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.215881:0:225777:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826338.215900:0:225777:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637056:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826338.215977:0:225777:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826338.221094:0:225777:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057c7539000: committing for last_committed 4294985437 gen 1 00000100:00100000:0.0:1787826338.221134:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637056:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826338.221152:0:225777:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221164:0:225777:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221174:0:225777:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221184:0:225777:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221202:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221209:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221215:0:225777:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826338.221259:0:225777:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158d880 x1874663332637056/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826338.221274:0:225777:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221288:0:225777:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7764 (0 0) 00000080:00200000:0.0:1787826338.221300:0:225777:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826338.221341:0:225777:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e54:0x0], updating i_size 4096 00000080:00200000:0.0:1787826338.221344:0:225777:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e54:0x0](ffff9057dfe0ec08): rc = 0 00000080:00010000:0.0:1787826338.221351:0:225777:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) for remote lock 0x95194e87608017eb 00000080:00010000:0.0:1787826338.221355:0:225777:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) for lock 0x95194e87608017e4 00000080:00010000:0.0:1787826338.221364:0:225777:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe0ec08) 00000080:00010000:0.0:1787826338.221367:0:225777:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017e4 from it ffffb512c7e3baa0 00010000:00010000:0.0:1787826338.221371:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221380:0:225777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3a00/0x95194e87608017e4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b719 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826338.221394:0:225777:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e87608017eb from it ffffb512c7e3baa0 00010000:00010000:0.0:1787826338.221397:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221404:0:225777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.221422:0:225777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), inode mode 41ed mask 1 00000080:00200000:0.0:1787826338.221443:0:225777:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826338.221455:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826338.221464:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.221473:0:225777:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.221479:0:225777:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826338.221483:0:225777:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826338.221485:0:225777:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608017dd 00000080:00010000:0.0:1787826338.221489:0:225777:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017dd from it ffffb512c7e3bbd0 00010000:00010000:0.0:1787826338.221491:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.221498:0:225777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826338.221505:0:225777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826338.221510:0:225777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=24 00000080:00200000:0.0:1787826338.221513:0:225777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), inode mode 41ed mask 1 00000080:00200000:0.0:1787826338.221524:0:225777:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), flags=0 00000080:00200000:0.0:1787826338.221528:0:225777:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), intent=getattr 00000002:00010000:0.0:1787826338.221540:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x0:0x0:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826338.221579:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221586:0:225777:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.221780:0:225777:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826338.230667:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637184:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826338.230692:0:225777:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230721:0:225777:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230730:0:225777:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230741:0:225777:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230752:0:225777:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230771:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230780:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230788:0:225777:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826338.230799:0:225777:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d880 x1874663332637184/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826338.230811:0:225777:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826338.230825:0:225777:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230846:0:225777:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7765 (0 0) 00000080:00200000:3.0:1787826338.230853:0:225777:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x198:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826338.230896:0:225777:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e55:0x0](ffff9057dfe0aa08): rc = 0 00000080:00200000:3.0:1787826338.230910:0:225777:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e55:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826338.230916:0:225777:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e55:0x0](ffff9057dfe0aa08) for lock 0x95194e87608017f2 00000080:00010000:3.0:1787826338.230925:0:225777:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e55:0x0](ffff9057dfe0aa08) 00000080:00010000:3.0:1787826338.230928:0:225777:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017f2 from it ffffb512c7e3bdb8 00010000:00010000:3.0:1787826338.230932:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.230946:0:225777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e87608017f2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b743 expref: -99 pid: 225777 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826338.231205:0:225777:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), flags=2048 00000080:00200000:3.0:1787826338.231210:0:225777:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e54:0x0](ffff9057dfe0ec08), intent=getattr 00000002:00010000:3.0:1787826338.231278:0:225777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826338.231332:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.231341:0:225777:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.231350:0:225777:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225777 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826338.231371:0:225777:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637312:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826338.231495:0:225777:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000100:00100000:3.0:1787826338.246112:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637312:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826338.246139:0:225777:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246172:0:225777:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246180:0:225777:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246196:0:225777:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246207:0:225777:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246773:0:225777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246787:0:225777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.246794:0:225777:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826338.246809:0:225777:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220000 x1874663332637312/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826338.246820:0:225777:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e87608017f9 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b75f expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826338.256449:0:225777:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9fd9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826338.279946:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43624 00000100:00100000:3.0:1787826338.279967:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:20167:x1874663346271488:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826338.279982:0:6159:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826338.280000:0:6159:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826338.280026:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@00000000cfa970bc[0x110, 2, [0x200000403:0x1dbe:0x0] hash] 00000020:00010000:3.0:1787826338.280029:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826333 atime: 1787826333 ctime: 1787826333 blocks: 8 00000020:00010000:3.0:1787826338.280032:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@00000000cb3e180a[0x110, 2, [0x280000401:0x521:0x0] hash] 00000020:00010000:3.0:1787826338.280034:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826333 atime: 1787826333 ctime: 1787826333 blocks: 8 00000100:00100000:3.0:1787826338.280089:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:20167:x1874663346271488:12345-192.168.203.156@tcp:106: Request processed in 135us (449us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826338.280097:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43624 00010000:00010000:0.0:1787826338.288382:0:2393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288408:0:2393:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288423:0:2393:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288430:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288437:0:2393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826338.288456:0:2393:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 6, setting rss=6; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826338.288462:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826338.288489:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288498:0:2393:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288507:0:2393:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288514:0:2393:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c200 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826338.288519:0:2393:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826338.288546:0:225775:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbe:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826338.288552:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:3.0:1787826338.288556:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826338.288561:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a610b0 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826338.288565:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826338.288568:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a610b0 (ffff9057d889c998/1) at cl_lock_fini():72 00010000:00010000:0.0:1787826338.288635:0:2393:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826338.288643:0:2393:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.288651:0:2393:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057d128a800 lock: 00000000935242d3/0x95194e8760801800 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826338.288672:0:225775:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787826338.288679:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332637824:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826338.288710:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332637952:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200008:3.0:1787826338.288774:0:225775:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbe:0x0], ppos: 0, count: 4194304 00000080:00200000:3.0:1787826338.288795:0:225775:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:3.0:1787826338.288809:0:225775:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:3.0:1787826338.288814:0:225775:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826338.288819:0:225775:0:(cl_io.c:214:cl_io_rw_init()) header@00000000c3fa0f31[0x110, 1, [0x200000403:0x1dbe:0x0] hash] 00000020:00200000:3.0:1787826338.289291:0:225775:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:3.0:1787826338.289298:0:225775:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00100000:3.0:1787826338.295252:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346271872 00000100:00100000:3.0:1787826338.295262:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346271872 00000100:00100000:3.0:1787826338.295328:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43625 00000100:00100000:3.0:1787826338.295338:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346271872:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826338.295355:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e87608017cf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b6da expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826338.295418:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826338.295424:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346271872:12345-192.168.203.156@tcp:104: Request processed in 89us (353us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826338.295431:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43625 00020000:00200000:0.0:1787826338.304066:0:225775:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826338.304075:0:225775:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826338.304100:0:225775:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826338.304119:0:225775:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826338.304124:0:225775:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5de30, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826338.304184:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.304198:0:225775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.304210:0:225775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826338.304257:0:225775:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638208:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826338.304348:0:225775:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826338.312493:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826338.312515:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346272256 00000100:00100000:3.0:1787826338.312525:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346272256 00000100:00100000:3.0:1787826338.312580:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346272768 00000100:00100000:3.0:1787826338.312584:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346272768 00000100:00100000:3.0:1787826338.312612:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43626 00000100:00100000:3.0:1787826338.312622:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346272256:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826338.312640:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826338.312746:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826338.312754:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346272256:12345-192.168.203.156@tcp:104: Request processed in 133us (571us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826338.312762:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43626 00000100:00100000:3.0:1787826338.312781:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43627 00000100:00100000:3.0:1787826338.312788:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c5063050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346272768:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826338.312799:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.312979:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826338.312987:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.312996:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b07a00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826338.313002:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400000000 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826338.313033:0:49149:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826338.313090:0:49149:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826338.313133:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 2/0,0 mode: --/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313146:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826338.313181:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 1/0,0 mode: --/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313189:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001a0be3a0/0x95194e876080179e lrc: 0/0,0 mode: --/PW res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4b655 expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313211:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826338.313215:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313248:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2de00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826338.313253:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313261:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313268:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826338.313288:0:49149:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbe:0x0]: losing layout lock 00010000:00010000:0.0:1787826338.313328:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 2/0,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313334:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826338.313341:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313347:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2de00/0x95194e87608017d6 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4b76d expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.313362:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826338.313364:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826338.313366:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826338.313384:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332638464:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826338.313448:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826338.313454:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332638336:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826338.317793:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1787826338.317812:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c5063050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346272768:12345-192.168.203.156@tcp:104: Request processed in 5018us (5446us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826338.317833:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43627 00010000:00010000:0.0:1787826338.317959:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826338.321672:0:225775:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638208:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826338.321699:0:225775:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.321714:0:225775:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.321723:0:225775:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826338.322154:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346272896 00000100:00100000:3.0:1787826338.322164:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346272896 00000100:00100000:3.0:1787826338.322346:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43628 00000100:00100000:3.0:1787826338.322359:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:125406:x1874663346272896:12345-192.168.203.156@tcp:105: 00010000:00010000:3.0:1787826338.322371:0:6159:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322381:0:6159:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322391:0:6159:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322402:0:6159:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322412:0:6159:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322423:0:6159:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x80000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826338.322453:0:6159:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9057d86d3800) 00000100:00100000:3.0:1787826338.322515:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:125406:x1874663346272896:12345-192.168.203.156@tcp:105: Request processed in 158us (1086us total) trans 0 rc 0/0 00010000:00010000:0.0:1787826338.322580:0:225775:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322592:0:225775:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322618:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826338.322631:0:225775:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322643:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322651:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826338.322677:0:225775:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000020:00010000:0.0:1787826338.322690:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826338.322693:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.322698:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5de30 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826338.322703:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322713:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826338.322728:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826338.322732:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5de30 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826338.322740:0:225775:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826338.322753:0:225775:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826338.322757:0:225775:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 1 00000080:00200000:0.0:1787826338.322760:0:225775:0:(file.c:2245:ll_file_io_generic()) f45j.sanityn: restart read from ppos=0 bytes=4194304 retries=999 ret=0: rc = 0 00000080:00200000:0.0:1787826338.322765:0:225775:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826338.322771:0:225775:0:(cl_io.c:214:cl_io_rw_init()) header@00000000c3fa0f31[0x110, 1, [0x200000403:0x1dbe:0x0] hash] 00000020:00200000:0.0:1787826338.322773:0:225775:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826338.322777:0:225775:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826338.322803:0:225775:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000403:0x1dbe:0x0](ffff9057dfe09988) 00000002:00010000:0.0:1787826338.322812:0:225775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dbe:0x0]) in obj [0x200000403:0x1dbe:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826338.322865:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322874:0:225775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.322883:0:225775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826338.322901:0:225775:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638592:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826338.322950:0:225775:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826338.333899:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43628 00000100:00100000:0.0:1787826338.334962:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332638464:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826338.337790:0:225777:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225777:1874663332637696:192.168.203.156@tcp:36:mrename.0 00000100:00100000:0.0:1787826338.340346:0:225775:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638592:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826338.340376:0:225775:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.340390:0:225775:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826338.340410:0:225775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158f480 x1874663332638592/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826354 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826338.340421:0:225775:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826338.340433:0:225775:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.340445:0:225775:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 16 res 8589935619/7614 (0 0) 00000080:00200000:0.0:1787826338.340451:0:225775:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000403:0x1dbe:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826338.340469:0:225775:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbe:0x0](ffff9057dfe09988) for lock 0x95194e876080180e 00000080:00010000:0.0:1787826338.340477:0:225775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080180e from it ffffb512c7d2baf8 00010000:00010000:0.0:1787826338.340481:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826338.340491:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340528:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340539:0:225775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826338.340548:0:225775:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000403:0x1dbe:0x0](00000000f57837cf) being reconfigured ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.340671:0:225775:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbe:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826338.340678:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340686:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826338.340710:0:225775:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826338.340714:0:225775:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826338.340734:0:225775:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826338.340749:0:225775:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826338.340753:0:225775:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d0b0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826338.340769:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340778:0:225775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340790:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340798:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340805:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826338.340817:0:225775:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826338.340846:0:225775:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbe:0x0] 00000008:00010000:0.0:1787826338.340853:0:225775:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826338.340854:0:225775:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5ddb0, osc lock ffff9057c8b838f0, flags 0x400001002 00010000:00010000:0.0:1787826338.340861:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340870:0:225775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826338.340887:0:225775:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826338.340890:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.340897:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826338.340903:0:225775:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826338.340909:0:225775:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbe:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826338.340912:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826338.340915:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.340918:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5ddb0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.340921:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826338.340924:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5ddb0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826338.340931:0:225775:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1787826338.340984:0:225775:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff9057c644b3c0 00000008:00010000:0.0:1787826338.341746:0:215335:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9057c644b3c0 00010000:00010000:0.0:1787826338.341774:0:215335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:0.0:1787826338.341810:0:215335:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158f480 - object 0x280000401:1313 offset 0<>4096 00000020:00010000:0.0:1787826338.350979:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826338.350983:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.350988:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d0b0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826338.351001:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826338.353637:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826338.353661:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826338.353665:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d0b0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826338.353678:0:225775:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826338.353694:0:225775:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826338.353699:0:225775:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826338.353704:0:225775:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:0.0:1787826338.353722:0:225775:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbe:0x0], ppos: 6, count: 4194304 00000080:00200000:0.0:1787826338.353734:0:225775:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:0.0:1787826338.353738:0:225775:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826338.353744:0:225775:0:(cl_io.c:214:cl_io_rw_init()) header@00000000c3fa0f31[0x110, 2, [0x200000403:0x1dbe:0x0] hash] 00000020:00200000:0.0:1787826338.353746:0:225775:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:0.0:1787826338.353750:0:225775:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826338.353766:0:225775:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:0.0:1787826338.353770:0:225775:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826338.353785:0:225775:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:0.0:1787826338.353803:0:225775:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826338.353808:0:225775:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d0b0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826338.353828:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353841:0:225775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353855:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353864:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353871:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826338.353892:0:225775:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:0.0:1787826338.353907:0:225775:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbe:0x0] 00000008:00010000:0.0:1787826338.353912:0:225775:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826338.353914:0:225775:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5ddb0, osc lock ffff9057c8b838f0, flags 0x400001002 00010000:00010000:0.0:1787826338.353922:0:225775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353930:0:225775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826338.353938:0:225775:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826338.353942:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.353949:0:225775:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826338.353963:0:225775:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826338.353970:0:225775:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbe:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826338.353972:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826338.353975:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.353978:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5ddb0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.353981:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89280 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826338.353983:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5ddb0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826338.353988:0:225775:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:0.0:1787826338.353996:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826338.353998:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826338.354000:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d0b0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826338.354004:0:225775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826338.354778:0:225775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000061adc995/0x95194e8760801807 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x521:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4b7a5 expref: -99 pid: 225775 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1787826338.354793:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826338.354796:0:225775:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d0b0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826338.354804:0:225775:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826338.354812:0:225775:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbe:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826338.354815:0:225775:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:0.0:1787826338.355093:0:225775:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:0.0:1787826338.355182:0:225775:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dbe:0x0](ffff9057dfe09988), flags 100000 00000002:00100000:0.0:1787826338.355689:0:225775:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb239880 x1874663332636672/t8589957436(8589957436) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826353 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826338.355700:0:225775:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826338.355705:0:225775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158fb80 x1874663332638976/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826338.355728:0:225775:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638976:192.168.203.156@tcp:35:cat.0 00000100:00100000:0.0:1787826338.355871:0:225775:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826338.361673:0:225775:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225775:1874663332638976:192.168.203.156@tcp:35:cat.0 00000002:00100000:0.0:1787826338.361708:0:225775:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb239880 x1874663332636672/t8589957436(8589957436) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826353 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826339.246875:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057d128a800: committing for last_committed 8589957438 gen 1 00000100:00100000:3.0:1787826339.246903:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332639360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.246922:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332639616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.246933:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957437 gen 1 00000100:00100000:3.0:1787826339.246973:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332639872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.246988:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332640128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.247011:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332639232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.247023:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332639488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.247036:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332639744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.247048:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332640000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826339.247058:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9057c7539000: committing for last_committed 4294976164 gen 19 00000100:00100000:3.0:1787826339.247069:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332640256:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826339.623245:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826339.626802:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826339.626846:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.626871:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.626891:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826339.628442:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826339.628457:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826339.628462:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608017b3 00000080:00010000:3.0:1787826339.628473:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017b3 from it ffffb512c7e8bb58 00010000:00010000:3.0:1787826339.628480:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.628494:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826339.628518:0:225813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826339.628532:0:225813:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:3.0:1787826339.628550:0:225813:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826339.628556:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348),name=d45j.sanityn 00000002:00010000:3.0:1787826339.628583:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e54:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.628597:0:225813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7764 (0 0) 00010000:00010000:3.0:1787826339.628665:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.628673:0:225813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.628689:0:225813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826339.628721:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640384:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826339.628841:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826339.642149:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640384:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826339.642186:0:225813:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642456:0:225813:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642469:0:225813:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642482:0:225813:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642494:0:225813:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642510:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642518:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642531:0:225813:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826339.642543:0:225813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223100 x1874663332640384/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826339.642557:0:225813:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642570:0:225813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7764 (0 0) 00000080:00200000:3.0:1787826339.642578:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826339.642621:0:225813:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e54:0x0], updating i_size 4096 00000080:00010000:3.0:1787826339.642629:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) for lock 0x95194e8760801815 00000080:00010000:3.0:1787826339.642635:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) 00000080:00010000:3.0:1787826339.642638:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801815 from it ffffb512c7e8bcb8 00010000:00010000:3.0:1787826339.642641:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.642650:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826339.642695:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.645518:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.645541:0:225813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.645562:0:225813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826339.645597:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640512:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826339.645711:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826339.652660:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640512:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826339.652695:0:225813:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 4/1,0 mode: --/CR res: [0x240000400:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.652709:0:225813:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.652722:0:225813:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.652734:0:225813:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653425:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653435:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653442:0:225813:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826339.653460:0:225813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223100 x1874663332640512/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826339.653474:0:225813:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653487:0:225813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/408 (0 0) 00010000:00010000:3.0:1787826339.653496:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653503:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826339.653540:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.653598:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653605:0:225813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.653615:0:225813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826339.653639:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640640:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826339.653736:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826339.663386:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640640:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826339.663420:0:225813:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x196:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663436:0:225813:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663450:0:225813:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663463:0:225813:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663477:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663485:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663493:0:225813:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826339.663505:0:225813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223100 x1874663332640640/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826339.663520:0:225813:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663534:0:225813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/406 (0 0) 00010000:00010000:3.0:1787826339.663543:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.663558:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826339.663662:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826339.663687:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.663696:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.663708:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826339.663715:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826339.663722:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826339.663726:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608017b3 00000080:00010000:3.0:1787826339.663730:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017b3 from it ffffb512c7e8bbf8 00010000:00010000:3.0:1787826339.663734:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.663807:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826339.664378:0:225813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826339.664388:0:225813:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:3.0:1787826339.664398:0:225813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348), inode mode 41ed mask 44 00000080:00200008:3.0:1787826339.664407:0:225813:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), flags 704400 00000002:00010000:3.0:1787826339.664429:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e54:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1787826339.664490:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.664498:0:225813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826339.664512:0:225813:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826339.664521:0:225813:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb223100 x1874663332640896/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826339.664531:0:225813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826339.664551:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640896:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826339.664774:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826339.667698:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332640896:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826339.667729:0:225813:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668142:0:225813:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668155:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668162:0:225813:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668171:0:225813:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668180:0:225813:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9000 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826339.668186:0:225813:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668193:0:225813:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668201:0:225813:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668208:0:225813:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826339.668215:0:225813:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.668344:0:225813:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080182a lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826339.668384:0:225813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb223100 x1874663332640896/t4294985440(4294985440) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787826339.668398:0:225813:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb223100 x1874663332640896/t4294985440(4294985440) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826339.668416:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826339.668461:0:225813:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e54:0x0], updating i_size 4096 00000002:00100000:3.0:1787826339.668486:0:225813:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb223100 x1874663332640896/t4294985440(4294985440) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1787826339.668586:0:225813:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826339.668595:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348),name=d45j.sanityn 00000002:00010000:3.0:1787826339.668620:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e54:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.668637:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668651:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826339.668663:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.668669:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668676:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668684:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668692:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826339.668709:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.668717:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668726:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.668733:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.669467:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826339.669486:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1787826339.669494:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) 00000080:00010000:3.0:1787826339.669498:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) for lock 0x95194e8760801815 00000080:00010000:3.0:1787826339.669504:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801815 from it ffffb512c7e8bcd8 00010000:00010000:3.0:1787826339.669507:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826339.669515:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:3.0:1787826339.669533:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.669543:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669552:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669560:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669567:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:3.0:1787826339.669576:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826339.669582:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669589:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669596:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826339.669602:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826339.669707:0:225813:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348) pos/size0/8192 32bit_api 0 00010000:00010000:3.0:1787826339.669724:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:3.0:1787826339.669764:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826339.669771:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826339.669780:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787826339.670713:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641024:192.168.203.156@tcp:37:rm.0 00000100:00100000:3.0:1787826339.670847:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826339.677884:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641024:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826339.678693:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826339.678716:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826339.678750:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826339.678770:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826339.678786:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826339.679449:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641152:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826339.679590:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c78743c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826339.690532:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641152:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826339.692377:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826339.692406:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826339.692599:0:225813:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), flags 704000 00000002:00100000:0.0:1787826339.692697:0:225813:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb223100 x1874663332640896/t4294985440(4294985440) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826355 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826339.692713:0:225813:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826339.692720:0:225813:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158fb80 x1874663332641408/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826339.692752:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641408:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826339.692886:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c78743c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826339.703921:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641408:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826339.703981:0:225813:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb223100 x1874663332640896/t4294985440(4294985440) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826339.704083:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826339.704107:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.704130:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.704149:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826339.704157:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826339.704163:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826339.704167:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e87608017b3 00000080:00010000:0.0:1787826339.704176:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608017b3 from it ffffb512c7e8bbf8 00010000:00010000:0.0:1787826339.704180:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.704190:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826339.704208:0:225813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826339.704216:0:225813:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826339.704281:0:225813:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348), inode mode 41ed mask 44 00000080:00200008:0.0:1787826339.704302:0:225813:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), flags 704400 00000002:00010000:0.0:1787826339.704323:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e54:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826339.704383:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.704393:0:225813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826339.704406:0:225813:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826339.704413:0:225813:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158fb80 x1874663332641664/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826339.704428:0:225813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826339.704450:0:225813:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641664:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826339.704680:0:225813:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c78743c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826339.711493:0:225813:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:225813:1874663332641664:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826339.711550:0:225813:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.711564:0:225813:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.711578:0:225813:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.711590:0:225813:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826339.711607:0:225813:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158fb80 x1874663332641664/t4294985442(4294985442) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826339.711621:0:225813:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158fb80 x1874663332641664/t4294985442(4294985442) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826339.711638:0:225813:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.711651:0:225813:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7764 (0 0) 00000080:00200000:0.0:1787826339.711659:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826339.711702:0:225813:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e54:0x0], updating i_size 4096 00000080:00010000:0.0:1787826339.711710:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) for lock 0x95194e8760801831 00000080:00010000:0.0:1787826339.711719:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801831 from it ffffb512c7e8bbd0 00010000:00010000:0.0:1787826339.711724:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826339.711735:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801831 lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e54:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b81c expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826339.711777:0:225813:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158fb80 x1874663332641664/t4294985442(4294985442) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826355 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826339.711874:0:225813:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e54:0x0](ffff9057dfe05348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826339.711882:0:225813:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e54:0x0](ffff9057dfe05348),name=d45j.sanityn 00000002:00010000:0.0:1787826339.711906:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e54:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.711917:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.711931:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826339.711941:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.711947:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.711955:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.711969:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.711976:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826339.711986:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.711994:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.712002:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.712010:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826339.712018:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826339.712028:0:225813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826339.712033:0:225813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) 00000080:00010000:0.0:1787826339.712037:0:225813:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e54:0x0](ffff9057dfe05348) for lock 0x95194e8760801815 00000080:00010000:0.0:1787826339.712041:0:225813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801815 from it ffffb512c7e8bcd8 00010000:00010000:0.0:1787826339.712044:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.712052:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e8760801815 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7e4 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826339.712065:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x198:0x0]) in obj [0x240000400:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.712071:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712079:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712086:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712093:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f9e00/0x95194e876080181c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b7f2 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826339.712100:0:225813:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x196:0x0]) in obj [0x200000401:0x196:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826339.712106:0:225813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712114:0:225813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712121:0:225813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826339.712127:0:225813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000100:00100000:0.0:1787826339.752801:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332642176:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826339.758969:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332642176:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826339.759907:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346275840 00000100:00100000:3.0:1787826339.759916:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346275840 00000100:00100000:3.0:1787826339.760018:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43630 00000100:00100000:3.0:1787826339.760029:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346275840:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826339.760053:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826339.760164:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826339.760172:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346275840:12345-192.168.203.156@tcp:104: Request processed in 145us (453us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826339.760180:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43630 00010000:00010000:0.0:1787826339.760251:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826339.760259:0:15672:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760268:0:15672:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826339.760274:0:15672:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760399:0:15672:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760409:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760539:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760547:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826339.760581:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760590:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801823 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x196:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b800 expref: -99 pid: 225813 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826339.760609:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826339.760611:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826339.760614:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826339.760623:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826339.760639:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332642560:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.765024:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332642560:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826339.785212:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346277504 00000100:00100000:3.0:1787826339.785813:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346277504 00000100:00100000:3.0:1787826339.785846:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43632 00000100:00100000:3.0:1787826339.785855:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346277504:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826339.785865:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826339.786007:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826339.786015:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346277504:12345-192.168.203.156@tcp:104: Request processed in 161us (869us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826339.786024:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43632 00010000:00010000:0.0:1787826339.786272:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826339.786281:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786291:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826339.786306:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786316:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786326:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786391:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786399:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826339.786426:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786434:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e87608017b3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b694 expref: -99 pid: 225775 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826339.786463:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826339.786467:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786475:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2000 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826339.786478:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786485:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786492:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786520:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786527:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826339.786533:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786539:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e87608017dd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b6fd expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826339.786555:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826339.786557:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826339.786559:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826339.786580:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332643328:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.789173:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826339.789201:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332643456:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.791858:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332643328:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.804201:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332643456:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826339.808961:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826339.810115:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332643584:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.814905:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332643584:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826339.815107:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826339.816052:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826339.816071:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332643712:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826339.819180:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332643712:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1787826339.915683:0:225821:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057c7539000) / proj=0 00000080:00000004:3.0:1787826339.915696:0:225821:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057c7539000: age 8149, max_age 8154 00800000:00000004:3.0:1787826339.915713:0:225821:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: age 8149, max_age 8154 00000100:00100000:3.0:1787826339.915772:0:225821:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:225821:1874663332643840:192.168.203.156@tcp:41:stat.0 00000100:00100000:3.0:1787826339.916664:0:225821:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826339.918601:0:225821:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:225821:1874663332643840:192.168.203.156@tcp:41:stat.0 00800000:00000004:3.0:1787826339.918638:0:225821:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: update ffff9057c6ef0450 cache blocks 1793357/1916558 objects 519507/534202 00000080:00000004:3.0:1787826339.918645:0:225821:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057c7539000: update ffff9057c6ef37c0 cache blocks 1793357/1916558 objects 519507/534202 00000080:00000004:3.0:1787826339.918650:0:225821:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793357/1916558 objects 519507/534202 00000100:00100000:3.0:1787826339.991911:0:225825:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225825:1874663332643968:192.168.203.156@tcp:101:lfs.0 00010000:00010000:3.0:1787826339.991992:0:225825:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992007:0:225825:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992021:0:225825:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992034:0:225825:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992048:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992056:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992064:0:225825:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826339.992076:0:225825:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0a80 x1874663332643968/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826339.992097:0:225825:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992114:0:225825:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826339.992122:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826339.992158:0:225825:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826339.992166:0:225825:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801846 00000080:00010000:3.0:1787826339.992172:0:225825:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826339.992175:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801846 from it ffffb512c7e8bb58 00010000:00010000:3.0:1787826339.992179:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992187:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826339.992255:0:225825:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826339.992283:0:225825:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:3.0:1787826339.992288:0:225825:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:3.0:1787826339.992325:0:225825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826339.992380:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992388:0:225825:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826339.992398:0:225825:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826339.992425:0:225825:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225825:1874663332644096:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826339.992518:0:225825:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826340.025011:0:225825:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225825:1874663332644096:192.168.203.156@tcp:101:lfs.0 00010000:00010000:3.0:1787826340.025044:0:225825:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025061:0:225825:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025069:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025077:0:225825:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025087:0:225825:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025093:0:225825:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9000 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826340.025098:0:225825:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025106:0:225825:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025113:0:225825:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025121:0:225825:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826340.025130:0:225825:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.025136:0:225825:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e876080184d lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826340.025158:0:225825:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057fb222a00 x1874663332644096/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826355 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826340.025175:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1787826340.025211:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826340.025304:0:225825:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826340.025316:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801846 from it ffffb512c7e8b930 00010000:00010000:3.0:1787826340.025320:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826340.025329:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826340.025441:0:225825:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826340.025459:0:225825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826340.025468:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826340.025476:0:225825:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826340.025488:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826340.025493:0:225825:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826340.025497:0:225825:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801846 00000080:00010000:3.0:1787826340.025502:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801846 from it ffffb512c7e8bbe8 00010000:00010000:3.0:1787826340.025506:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826340.025514:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826340.025524:0:225825:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826340.025532:0:225825:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:3.0:1787826340.025563:0:225825:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826340.025569:0:225825:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826340.025577:0:225825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826340.025583:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826340.025592:0:225825:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826340.025598:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826340.025602:0:225825:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826340.025605:0:225825:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801846 00000080:00010000:3.0:1787826340.025609:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801846 from it ffffb512c7e8bcb8 00010000:00010000:3.0:1787826340.025613:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826340.025621:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826340.026323:0:225825:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826340.026330:0:225825:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826340.026345:0:225825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826340.026404:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.026413:0:225825:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.026437:0:225825:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225825 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826340.026465:0:225825:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:225825:1874663332644224:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826340.026567:0:225825:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826340.031259:0:225825:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:225825:1874663332644224:192.168.203.156@tcp:101:lfs.0 00010000:00010000:3.0:1787826340.031312:0:225825:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031327:0:225825:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031340:0:225825:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031355:0:225825:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031369:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031378:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031386:0:225825:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826340.031398:0:225825:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222a00 x1874663332644224/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826340.031412:0:225825:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031856:0:225825:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826340.031865:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826340.031903:0:225825:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826340.031911:0:225825:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801854 00000080:00010000:3.0:1787826340.031917:0:225825:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826340.031920:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801854 from it ffffb512c7e8bcb8 00010000:00010000:3.0:1787826340.031924:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.031940:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826340.032356:0:225825:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826340.032382:0:225825:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826340.032396:0:225825:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826340.032409:0:225825:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1787826340.032416:0:225825:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826340.032435:0:225825:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826340.032438:0:225825:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801846 00000080:00010000:3.0:1787826340.032443:0:225825:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801846 from it ffffb512c7e8bbb0 00010000:00010000:3.0:1787826340.032447:0:225825:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1787826340.032455:0:225825:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1787826340.032465:0:225825:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 44 00000080:00200008:3.0:1787826340.032476:0:225825:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200080:3.0:1787826340.032499:0:225825:0:(dir.c:2001:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) cmd=c03066f0 arg=7ffddb021bf0 00000080:00200000:3.0:1787826340.032518:0:225825:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) name=d45j.sanityn stripe_offset=1 stripe_count=2, hash_type=4 00000020:00100000:3.0:1787826340.032589:0:225825:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826340.032599:0:225825:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222a00 x1874663332644480/t0(0) o36->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826340.032626:0:225825:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225825:1874663332644480:192.168.203.156@tcp:36:lfs.0 00000100:00100000:3.0:1787826340.032910:0:225825:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00010000:00010000:0.0:1787826340.060121:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826340.060134:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060149:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826340.060154:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060164:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060180:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060311:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060319:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826340.060346:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060353:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9400/0x95194e8760801854 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8fc expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.060380:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826340.060384:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060391:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826340.060394:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060402:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060409:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060694:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060707:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826340.060714:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060722:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801846 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4b8d9 expref: -99 pid: 225825 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826340.060736:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.060738:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.060739:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826340.060747:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826340.060757:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332644736:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826340.070470:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332644736:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826340.080193:0:225825:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:225825:1874663332644480:192.168.203.156@tcp:36:lfs.0 00000080:00200000:3.0:1787826340.080417:0:225825:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448) 00000080:00200000:3.0:1787826340.080425:0:225825:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 4096 00000080:00200008:3.0:1787826340.080465:0:225825:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000100:00100000:3.0:1787826340.086352:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332644864:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826340.086387:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086400:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086412:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086423:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086436:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086444:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086456:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826340.086467:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de1c00 x1874663332644864/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826340.086481:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086496:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826340.086504:0:225814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826340.086537:0:225814:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826340.086544:0:225814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080185b 00000080:00010000:3.0:1787826340.086550:0:225814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826340.086554:0:225814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080185b from it ffffb512c7e83bf8 00010000:00010000:3.0:1787826340.086557:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086566:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826340.086595:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826340.086617:0:225814:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:3.0:1787826340.086622:0:225814:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826340.086647:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826340.086696:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086703:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826340.086712:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826340.086735:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332644992:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826340.086832:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.097316:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332644992:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.097351:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097377:0:225814:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097386:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097398:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097416:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097430:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097439:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097447:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.097460:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb221f80 x1874663332644992/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.097473:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097485:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7766 (0 0) 00000002:00010000:0.0:1787826340.097505:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e56:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826340.097564:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097573:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.097583:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.097612:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645120:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.097698:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.101362:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645120:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.101400:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101414:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101427:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101439:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101452:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101461:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101470:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.101482:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158dc00 x1874663332645120/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.101503:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101518:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7766 (0 0) 00000080:00200000:0.0:1787826340.101543:0:225814:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1787826340.101632:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826340.101688:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101696:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.101706:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.101729:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645248:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.101816:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.104431:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645248:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.104460:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104475:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104488:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104506:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104520:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104529:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104537:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.104549:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158df80 x1874663332645248/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.104624:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104639:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/409 (0 0) 00010000:00010000:0.0:1787826340.104648:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104658:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826340.104693:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826340.104746:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104755:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.104764:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.104784:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645376:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.104954:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.110341:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645376:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.110379:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110395:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110409:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110422:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110435:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110444:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110457:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.110470:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158df80 x1874663332645376/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.110484:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110500:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/407 (0 0) 00010000:00010000:0.0:1787826340.110510:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110520:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.110577:0:225814:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 8192 00000080:00200000:0.0:1787826340.110582:0:225814:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e56:0x0](ffff9057dfdef448): rc = 0 00000080:00010000:0.0:1787826340.110594:0:225814:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) for remote lock 0x95194e8760801869 00000080:00010000:0.0:1787826340.110599:0:225814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) for lock 0x95194e8760801862 00000080:00010000:0.0:1787826340.110615:0:225814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) 00000080:00010000:0.0:1787826340.110619:0:225814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801862 from it ffffb512c7e83ac8 00010000:00010000:0.0:1787826340.110623:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110630:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826340.110639:0:225814:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801869 from it ffffb512c7e83ac8 00010000:00010000:0.0:1787826340.110643:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110652:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.110678:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:0.0:1787826340.110707:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 3 00000080:00200000:0.0:1787826340.110714:0:225814:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), file ffff9057d05f9080, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826340.110723:0:225814:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=open|creat 00000002:00010000:0.0:1787826340.110754:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x200000403:0x1dbf:0x0]) in obj [0x200000401:0x197:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826340.110767:0:225814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.110835:0:225814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.110843:0:225814:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826340.110845:0:225814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801877 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b988 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.110888:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/0,1 mode: --/CW res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.110895:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/0,1 mode: --/CW res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826340.110907:0:225814:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826340.110915:0:225814:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158dc00 x1874663332645632/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826340.110925:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/0,1 mode: --/CW res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.110949:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645632:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.111143:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.116061:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645632:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.116099:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/0,1 mode: --/CW res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116138:0:225814:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116159:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116172:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116183:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116197:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116209:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116218:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.116273:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158dc00 x1874663332645632/t8589957443(8589957443) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826340.116286:0:225814:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158dc00 x1874663332645632/t8589957443(8589957443) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.116311:0:225814:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826340.116326:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116338:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7615 (0 0) 00000080:00200000:0.0:1787826340.116345:0:225814:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000401:0x197:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826340.116362:0:225814:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=open|creat 00000080:00200000:0.0:1787826340.116538:0:225814:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dbf:0x0](ffff9057dfe07448) 00000080:00200000:0.0:1787826340.116798:0:225814:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826340.116807:0:225814:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000403:0x1dbf:0x0](ffff9057dfe07448) 00000080:00010000:0.0:1787826340.116811:0:225814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbf:0x0](ffff9057dfe07448) for lock 0x95194e876080187e 00000080:00010000:0.0:1787826340.116825:0:225814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbf:0x0](ffff9057dfe07448) 00000080:00200008:0.0:1787826340.116841:0:225814:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1dbf:0x0](ffff9057dfe07448), flags 101101 00000002:00100000:0.0:1787826340.116878:0:225814:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158dc00 x1874663332645632/t8589957443(8589957443) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826340.116914:0:225814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080187e from it ffff9057f9ad4c00 00010000:00010000:0.0:1787826340.116922:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.116935:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.116970:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1dbf:0x0](ffff9057dfe07448), inode mode 81a4 mask 40 00000080:00200008:0.0:1787826340.117043:0:225814:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn:[0x200000403:0x1dbf:0x0], ppos: 0, count: 6 00000080:00200000:0.0:1787826340.117054:0:225814:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826340.117059:0:225814:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:0.0:1787826340.117062:0:225814:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: write ppos: 0, bytes: 6 00000080:00200000:0.0:1787826340.117067:0:225814:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826340.117074:0:225814:0:(cl_io.c:214:cl_io_rw_init()) header@0000000092cdb071[0x110, 1, [0x200000403:0x1dbf:0x0] hash] 00000020:00200000:0.0:1787826340.117077:0:225814:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:0.0:1787826340.117081:0:225814:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbf:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826340.117095:0:225814:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787826340.117103:0:225814:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:0.0:1787826340.117107:0:225814:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826340.117125:0:225814:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787826340.117140:0:225814:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826340.117144:0:225814:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5ddb0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826340.117196:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.117205:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.117216:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.117282:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645760:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.117382:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.120791:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332645760:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.120819:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.120839:0:225814:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.120851:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.120865:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826340.120878:0:225814:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.120890:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.120898:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826340.120929:0:225814:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn: write [0, 6) append 0 append lockpos 0 00000080:00200000:0.0:1787826340.120948:0:225814:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826340.121001:0:225814:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:0.0:1787826340.121019:0:225814:0:(rw26.c:929:ll_write_end()) page@00000000e2d1d6d2[3 000000009703728f 2 1 0000000070454659 0 0 0] 00000080:00200000:0.0:1787826340.121023:0:225814:0:(rw26.c:929:ll_write_end()) vmpage @000000008481b2f8 17ffffc0001001 3:0 ffff9057fb2ba9a0 0 lru 00000080:00200000:0.0:1787826340.121030:0:225814:0:(rw26.c:929:ll_write_end()) osc-page@00000000a75a2261 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009958d8f2 000000005df1c56b > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787826340.121032:0:225814:0:(rw26.c:929:ll_write_end()) end page@00000000e2d1d6d2 00000080:00200000:0.0:1787826340.121033:0:225814:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787826340.121038:0:225814:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:0.0:1787826340.121057:0:225814:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005746e05d ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826340.121076:0:225814:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfe075f8, count 1, dirtied 1 00000080:00200000:0.0:1787826340.121080:0:225814:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:0.0:1787826340.121083:0:225814:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn: write bytes 0, result: 6 00000020:00010000:0.0:1787826340.121098:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826340.121101:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826340.121106:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5ddb0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826340.121110:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.121120:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826340.121137:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826340.121140:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5ddb0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826340.121147:0:225814:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826340.121155:0:225814:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787826340.121164:0:225814:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826340.121168:0:225814:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826340.121175:0:225814:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbf:0x0] updating i_size 6 i_blocks 0 00000080:00200000:0.0:1787826340.121180:0:225814:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:0.0:1787826340.121209:0:225814:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dbf:0x0](ffff9057dfe07448), flags 100001 00000002:00100000:0.0:1787826340.121300:0:225814:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158dc00 x1874663332645632/t8589957443(8589957443) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826340.121312:0:225814:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826340.121318:0:225814:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158df80 x1874663332646016/t0(0) o35->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826340.121338:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646016:192.168.203.156@tcp:35:bash.0 00000100:00100000:0.0:1787826340.121411:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.124159:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646016:192.168.203.156@tcp:35:bash.0 00000002:00100000:0.0:1787826340.124195:0:225814:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158dc00 x1874663332645632/t8589957443(8589957443) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826340.124385:0:225814:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826340.124403:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826340.124422:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.124435:0:225814:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.124444:0:225814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826340.124450:0:225814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826340.124454:0:225814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080185b 00000080:00010000:0.0:1787826340.124462:0:225814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080185b from it ffffb512c7e83bf8 00010000:00010000:0.0:1787826340.124466:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.124475:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826340.124488:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826340.124495:0:225814:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=25 00000080:00200000:0.0:1787826340.124501:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:0.0:1787826340.124513:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 3 00000080:00200000:0.0:1787826340.124518:0:225814:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), file ffff9057d886c780, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826340.124526:0:225814:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=open|creat 00000002:00010000:0.0:1787826340.124545:0:225814:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x240000403:0x1e57:0x0]) in obj [0x240000400:0x199:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826340.124559:0:225814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.124678:0:225814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.124793:0:225814:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826340.124797:0:225814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801870 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b97a expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.124832:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/0,1 mode: --/CW res: [0x240000400:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.124839:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/0,1 mode: --/CW res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826340.124852:0:225814:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826340.124860:0:225814:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ca80 x1874663332646272/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826340.124869:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/0,1 mode: --/CW res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.124889:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646272:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.125873:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.132377:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646272:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.132413:0:225814:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/0,1 mode: --/CW res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132449:0:225814:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132458:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132470:0:225814:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132482:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132497:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132511:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132520:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826340.132533:0:225814:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158ca80 x1874663332646272/t4294985447(4294985447) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826340.132548:0:225814:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ca80 x1874663332646272/t4294985447(4294985447) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826340.132573:0:225814:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826340.132589:0:225814:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.132600:0:225814:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7767 (0 0) 00000080:00200000:0.0:1787826340.132608:0:225814:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000400:0x199:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826340.132624:0:225814:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=open|creat 00000080:00200000:0.0:1787826340.132665:0:225814:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e57:0x0](ffff9057dfe04b08) 00000080:00200000:0.0:1787826340.132764:0:225814:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e57:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826340.132770:0:225814:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe04b08) 00000080:00010000:0.0:1787826340.132773:0:225814:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe04b08) for lock 0x95194e876080188c 00000080:00010000:0.0:1787826340.132785:0:225814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe04b08) 00000080:00200008:0.0:1787826340.132803:0:225814:0:(file.c:1090:ll_file_open()) START file f45j.sanityn-2:[0x240000403:0x1e57:0x0](ffff9057dfe04b08), flags 101101 00000002:00100000:0.0:1787826340.133727:0:225814:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ca80 x1874663332646272/t4294985447(4294985447) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826340.133778:0:225814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080188c from it ffff9057f9ad4c00 00010000:00010000:0.0:1787826340.133787:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.133798:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.133979:0:225814:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e57:0x0](ffff9057dfe04b08), inode mode 81a4 mask 40 00000080:00200008:0.0:1787826340.134115:0:225814:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn-2:[0x240000403:0x1e57:0x0], ppos: 0, count: 6 00000080:00200000:0.0:1787826340.134130:0:225814:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826340.134137:0:225814:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:0.0:1787826340.134144:0:225814:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn-2: write ppos: 0, bytes: 6 00000080:00200000:0.0:1787826340.134149:0:225814:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn-2: desiginated mirror: 0 00000020:00200000:0.0:1787826340.134157:0:225814:0:(cl_io.c:214:cl_io_rw_init()) header@000000007012cf63[0x110, 1, [0x240000403:0x1e57:0x0] hash] 00000020:00200000:0.0:1787826340.134160:0:225814:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:0.0:1787826340.134165:0:225814:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e57:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826340.134180:0:225814:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787826340.134188:0:225814:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:0.0:1787826340.134191:0:225814:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826340.134213:0:225814:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787826340.134488:0:225814:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826340.134495:0:225814:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d6b0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826340.134554:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.134575:0:225814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.134588:0:225814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826340.134614:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646400:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826340.134716:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.137094:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646400:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826340.137126:0:225814:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137149:0:225814:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137162:0:225814:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137178:0:225814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826340.137191:0:225814:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137205:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137213:0:225814:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826340.137277:0:225814:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn-2: write [0, 6) append 0 append lockpos 0 00000080:00200000:0.0:1787826340.137308:0:225814:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826340.137372:0:225814:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:0.0:1787826340.137384:0:225814:0:(rw26.c:929:ll_write_end()) page@00000000e8e80bc7[3 00000000a00c95d4 2 1 0000000070454659 0 0 0] 00000080:00200000:0.0:1787826340.137388:0:225814:0:(rw26.c:929:ll_write_end()) vmpage @000000003b2305b5 17ffffc0001001 3:0 ffff9057fb2ba8c0 0 lru 00000080:00200000:0.0:1787826340.137396:0:225814:0:(rw26.c:929:ll_write_end()) osc-page@00000000c1a2a090 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 0000000072dd0513 00000000e98fa6c5 > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787826340.137399:0:225814:0:(rw26.c:929:ll_write_end()) end page@00000000e8e80bc7 00000080:00200000:0.0:1787826340.137400:0:225814:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787826340.137405:0:225814:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:0.0:1787826340.137428:0:225814:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826340.137449:0:225814:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfe04cb8, count 1, dirtied 1 00000080:00200000:0.0:1787826340.137455:0:225814:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:0.0:1787826340.137458:0:225814:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn-2: write bytes 0, result: 6 00000020:00010000:0.0:1787826340.137475:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_release():245 00000020:00010000:0.0:1787826340.137479:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826340.137483:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d6b0 (ffff9057f989c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826340.137488:0:225814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826340.137498:0:225814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826340.137521:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2b78 (ffff9057d889c998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826340.137524:0:225814:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d6b0 (ffff9057d889c998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826340.137533:0:225814:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826340.137542:0:225814:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787826340.137551:0:225814:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e57:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826340.137556:0:225814:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn-2: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826340.137575:0:225814:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e57:0x0] updating i_size 6 i_blocks 0 00000080:00200000:0.0:1787826340.137581:0:225814:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:0.0:1787826340.137624:0:225814:0:(file.c:479:ll_file_release()) START file f45j.sanityn-2:[0x240000403:0x1e57:0x0](ffff9057dfe04b08), flags 100001 00000002:00100000:0.0:1787826340.137696:0:225814:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158ca80 x1874663332646272/t4294985447(4294985447) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826340.137709:0:225814:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826340.137714:0:225814:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f800 x1874663332646656/t0(0) o35->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826340.137738:0:225814:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646656:192.168.203.156@tcp:35:bash.0 00000100:00100000:0.0:1787826340.137804:0:225814:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826340.140216:0:225814:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:225814:1874663332646656:192.168.203.156@tcp:35:bash.0 00000002:00100000:0.0:1787826340.140277:0:225814:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158ca80 x1874663332646272/t4294985447(4294985447) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826356 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826340.157385:0:225828:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:0.0:1787826340.158428:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 4 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1787826340.158442:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826340.158496:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.158514:0:34880:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbf:0x0]: losing layout lock 00010000:00010000:0.0:1787826340.158544:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.158611:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826340.158619:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.158626:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.158632:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1787826340.158662:0:34880:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_07:fc0b407b-a541-420d-8712-b9c8da18fe8c:34880:1874663332646784:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826340.158772:0:34880:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00010000:00010000:3.0:1787826340.160979:0:225828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787826340.161848:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826340.161864:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826340.161901:0:34879:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e57:0x0]: losing layout lock 00010000:00010000:0.0:1787826340.161953:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.161968:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.162014:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.162022:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.162029:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.162035:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1787826340.162063:0:34879:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_06:fc0b407b-a541-420d-8712-b9c8da18fe8c:34879:1874663332647040:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826340.162169:0:34879:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b519c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826340.166630:0:6167:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:398e0287-d3c7-44dd-b1d9-6b85f698e615:6167:1874663332646912:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826340.166668:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e876080180e lrc: 0/0,0 mode: --/CR res: [0x200000403:0x1dbe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b7c8 expref: -99 pid: 225775 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826340.166722:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826340.166725:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826340.166727:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826340.166784:0:34879:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_06:fc0b407b-a541-420d-8712-b9c8da18fe8c:34879:1874663332647040:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826340.166800:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e876080188c lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b9ce expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.166834:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801869 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b96c expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.166847:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e87608017eb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e54:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b727 expref: -99 pid: 225777 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.166870:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.166872:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.166874:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826340.166895:0:34880:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_07:fc0b407b-a541-420d-8712-b9c8da18fe8c:34880:1874663332646784:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826340.166906:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06e00/0x95194e876080185b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b942 expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826340.166921:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e876080187e lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4b9a4 expref: -99 pid: 225814 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.166939:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801862 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4b95e expref: -99 pid: 225814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826340.166960:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.166962:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.166963:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826340.644803:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826340.774833:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826340.774855:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826340.774892:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787826342.087408:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346285312 00000100:00100000:3.0:1787826342.087421:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346285312 00000100:00100000:3.0:1787826342.087503:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43637 00000100:00100000:3.0:1787826342.087517:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8759:x1874663346285312:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826342.087542:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e87608017ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4b67f expref: -99 pid: 225594 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826342.087659:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826342.087667:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8759:x1874663346285312:12345-192.168.203.156@tcp:104: Request processed in 153us (660us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826342.087674:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43637 00010000:00010000:0.0:1787826343.927361:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.927387:0:225995:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.927401:0:225995:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826343.927438:0:225995:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332647680:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826343.927578:0:225995:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826343.931946:0:225995:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332647680:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826343.931989:0:225995:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 4/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932003:0:225995:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 4/1,0 mode: --/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932017:0:225995:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932035:0:225995:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932050:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932058:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932065:0:225995:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826343.932078:0:225995:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332647680/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826359 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826343.932093:0:225995:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932107:0:225995:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/409 (0 0) 00010000:00010000:0.0:1787826343.932116:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932125:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e87608018af lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba29 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826343.932158:0:225995:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826343.932217:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932465:0:225995:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.932477:0:225995:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826343.932516:0:225995:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332647808:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826343.932623:0:225995:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826343.935308:0:225995:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332647808:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826343.935340:0:225995:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935354:0:225995:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935367:0:225995:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935379:0:225995:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935392:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935400:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935413:0:225995:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826343.935426:0:225995:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332647808/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826359 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826343.935439:0:225995:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935453:0:225995:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/407 (0 0) 00010000:00010000:0.0:1787826343.935462:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935471:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826343.935629:0:225995:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 8192 00000080:00010000:0.0:1787826343.935645:0:225995:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) for remote lock 0x95194e87608018a8 00000080:00010000:0.0:1787826343.935651:0:225995:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) for lock 0x95194e87608018a1 00000080:00010000:0.0:1787826343.935665:0:225995:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) 00000080:00010000:0.0:1787826343.935669:0:225995:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018a1 from it ffffb512c7f23ac8 00010000:00010000:0.0:1787826343.935674:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608018a1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba0d expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935682:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608018a1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba0d expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826343.935696:0:225995:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e87608018a8 from it ffffb512c7f23ac8 00010000:00010000:0.0:1787826343.935701:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e87608018a8 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba1b expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935710:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e87608018a8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba1b expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826343.935738:0:225995:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787826343.935766:0:225995:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), file ffff9057c56d6000, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1787826343.935773:0:225995:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), intent=open 00000002:00010000:0.0:1787826343.935800:0:225995:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x197:0x0], intent: open flags 0100001 00010000:00010000:0.0:1787826343.935860:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826343.935868:0:225995:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826343.935880:0:225995:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826343.936050:0:225995:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332648064/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826343.936060:0:225995:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225995 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826343.936085:0:225995:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332648064:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826343.936508:0:225995:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.354744:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332648192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.354884:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332648448:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.354915:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332648704:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.354938:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332648960:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.354961:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332649216:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.354991:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:3.0:1787826344.354998:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332648320:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.355018:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332648576:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.355036:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332648832:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826344.355055:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332649088:192.168.203.156@tcp:400:kworker.0 00000080:00200000:3.0:1787826344.414384:0:225997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826344.414426:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826344.414506:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.414519:0:225997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.414532:0:225997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.414566:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649344:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826344.415041:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000080:00200000:3.0:1787826344.436653:0:225997:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826344.436683:0:225997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608018c4 00000080:00010000:3.0:1787826344.436695:0:225997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826344.436698:0:225997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018c4 from it ffffb512c7ff3bd0 00010000:00010000:3.0:1787826344.436706:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.436722:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.436762:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826344.437121:0:225997:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=24 00000080:00200000:3.0:1787826344.437128:0:225997:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826344.437161:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826344.437258:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.437277:0:225997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.437612:0:225997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.437649:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649472:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826344.437767:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.443492:0:225997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649472:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826344.443798:0:225997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.443833:0:225997:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.443841:0:225997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.443854:0:225997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.443865:0:225997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.443881:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.444460:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.444475:0:225997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826344.444498:0:225997:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb222680 x1874663332649472/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826360 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826344.444511:0:225997:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.444524:0:225997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7766 (0 0) 00000002:00010000:3.0:1787826344.444985:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e56:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826344.445352:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.445361:0:225997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.445372:0:225997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.445399:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649600:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826344.445508:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.451864:0:225997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649600:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826344.451911:0:225997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451926:0:225997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451939:0:225997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451952:0:225997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451965:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451974:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.451981:0:225997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826344.451994:0:225997:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb223480 x1874663332649600/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826360 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826344.452009:0:225997:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.452025:0:225997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7766 (0 0) 00000080:00200000:3.0:1787826344.452061:0:225997:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826344.452106:0:225997:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 4096 00000080:00200000:3.0:1787826344.452110:0:225997:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e56:0x0](ffff9057dfdef448): rc = 0 00000080:00010000:3.0:1787826344.452120:0:225997:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) for remote lock 0x95194e87608018d2 00000080:00010000:3.0:1787826344.452125:0:225997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) for lock 0x95194e87608018cb 00000080:00010000:3.0:1787826344.452138:0:225997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfdef448) 00000080:00010000:3.0:1787826344.452141:0:225997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018cb from it ffffb512c7ff3aa0 00010000:00010000:3.0:1787826344.452146:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.452156:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e87608018cb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba76 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826344.452172:0:225997:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e87608018d2 from it ffffb512c7ff3aa0 00010000:00010000:3.0:1787826344.452176:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.452183:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8400/0x95194e87608018d2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba84 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.452208:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:3.0:1787826344.452301:0:225997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826344.452320:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826344.452331:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.452343:0:225997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.452349:0:225997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826344.452354:0:225997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826344.452358:0:225997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e87608018c4 00000080:00010000:3.0:1787826344.452361:0:225997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018c4 from it ffffb512c7ff3bd0 00010000:00010000:3.0:1787826344.452365:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.452377:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826344.452386:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826344.452391:0:225997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=24 00000080:00200000:3.0:1787826344.452398:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 1 00000080:00200000:3.0:1787826344.452414:0:225997:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), flags=0 00000080:00200000:3.0:1787826344.452420:0:225997:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=getattr 00000002:00010000:3.0:1787826344.452435:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x0:0x0:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826344.452484:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.452493:0:225997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.452503:0:225997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.452539:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649728:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826344.452626:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.459821:0:225997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649728:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826344.459857:0:225997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.459905:0:225997:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.459921:0:225997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.459933:0:225997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.459945:0:225997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460316:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460326:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460334:0:225997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826344.460348:0:225997:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223480 x1874663332649728/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826360 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826344.460360:0:225997:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826344.460374:0:225997:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460390:0:225997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7767 (0 0) 00000080:00200000:3.0:1787826344.460398:0:225997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x199:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826344.460459:0:225997:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e57:0x0](ffff9057dfe04b08): rc = 0 00000080:00200000:3.0:1787826344.460474:0:225997:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e57:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826344.460480:0:225997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe04b08) for lock 0x95194e87608018d9 00000080:00010000:3.0:1787826344.460492:0:225997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe04b08) 00000080:00010000:3.0:1787826344.460495:0:225997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018d9 from it ffffb512c7ff3db8 00010000:00010000:3.0:1787826344.460498:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460506:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.460755:0:225997:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), flags=2048 00000080:00200000:3.0:1787826344.460761:0:225997:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), intent=getattr 00000002:00010000:3.0:1787826344.460787:0:225997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826344.460842:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460850:0:225997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.460861:0:225997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 225997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.460886:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649856:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826344.461412:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.475380:0:225997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332649856:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826344.475415:0:225997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475452:0:225997:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475459:0:225997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475471:0:225997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475483:0:225997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475495:0:225997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475503:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475510:0:225997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826344.475523:0:225997:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223480 x1874663332649856/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826360 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826344.475546:0:225997:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826344.475561:0:225997:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475574:0:225997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7615 (0 0) 00000080:00200000:3.0:1787826344.475581:0:225997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x197:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826344.475636:0:225997:0:(namei.c:147:ll_iget()) got inode: [0x200000403:0x1dbf:0x0](ffff9057dfe07448): rc = 0 00000080:00200000:3.0:1787826344.475651:0:225997:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dbf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826344.475656:0:225997:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dbf:0x0](ffff9057dfe07448) for lock 0x95194e87608018e0 00000080:00010000:3.0:1787826344.475668:0:225997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dbf:0x0](ffff9057dfe07448) 00000080:00010000:3.0:1787826344.475672:0:225997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018e0 from it ffffb512c7ff3db8 00010000:00010000:3.0:1787826344.475675:0:225997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.475684:0:225997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.475721:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 3 00000080:00200000:3.0:1787826344.475728:0:225997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfdef448), inode mode 41ed mask 3 00000080:00200000:3.0:1787826344.475738:0:225997:0:(namei.c:2492:ll_rename()) VFS Op:oldname=f45j.sanityn-2, src_dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448), newname=f45j.sanityn, tgt_dir=[0x240000403:0x1e56:0x0](ffff9057dfdef448) 00010000:00010000:3.0:1787826344.475776:0:225997:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.475804:0:225997:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e57:0x0]: losing layout lock 00010000:00010000:3.0:1787826344.475847:0:225997:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.475858:0:225997:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826344.476297:0:225997:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e87608018d9 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4baa0 expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.476329:0:225997:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826344.476341:0:225997:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbf:0x0]: losing layout lock 00010000:00010000:3.0:1787826344.476367:0:225997:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.476374:0:225997:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826344.476376:0:225997:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e87608018e0 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4babc expref: -99 pid: 225997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1787826344.476398:0:225997:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787826344.476407:0:225997:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222680 x1874663332650240/t0(0) o36->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:13/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787826344.476430:0:225997:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332650240:192.168.203.156@tcp:36:mrename.0 00000100:00100000:3.0:1787826344.476515:0:225997:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826344.483117:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332649984:192.168.203.156@tcp:103:mrename.0 00000100:00100000:0.0:1787826344.487023:0:225995:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332648064:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826344.487652:0:225995:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487688:0:225995:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487698:0:225995:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487710:0:225995:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487724:0:225995:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487736:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487744:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826344.487751:0:225995:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826344.487766:0:225995:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057f158c700 x1874663332648064/t8589957449(8589957449) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826359 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826344.490207:0:225995:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158c700 x1874663332648064/t8589957449(8589957449) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826359 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826344.490253:0:225995:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.497129:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346287616 00000100:00100000:3.0:1787826344.497139:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346287616 00000100:00100000:3.0:1787826344.497300:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43639 00000100:00100000:3.0:1787826344.497312:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346287616:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826344.497333:0:6159:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.497348:0:6159:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826344.497384:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@0000000092cdb071[0x110, 2, [0x200000403:0x1dbf:0x0] hash] 00000020:00010000:3.0:1787826344.497386:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826340 atime: 1787826340 ctime: 1787826340 blocks: 1 00000020:00010000:3.0:1787826344.497390:0:6159:0:(cl_object.c:322:cl_object_glimpse()) header@00000000f09ceae7[0x110, 3, [0x2c0000401:0x522:0x0] hash] 00000020:00010000:3.0:1787826344.497391:0:6159:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826340 atime: 1787826340 ctime: 1787826340 blocks: 1 00000100:00100000:3.0:1787826344.497462:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:35097:x1874663346287616:12345-192.168.203.156@tcp:106: Request processed in 151us (446us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826344.497474:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43639 00000100:00100000:3.0:1787826344.498499:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346287744 00000100:00100000:3.0:1787826344.498507:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346287744 00000100:00100000:3.0:1787826344.498551:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43640 00000100:00100000:3.0:1787826344.498561:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346287744:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826344.498576:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.498640:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826344.498647:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346287744:12345-192.168.203.156@tcp:104: Request processed in 88us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826344.498653:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43640 00010000:00010000:0.0:1787826344.498701:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826344.498712:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.498723:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3000 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826344.498728:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.498739:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.498749:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.498824:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.498832:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826344.500501:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.500520:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e87608018b6 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4ba37 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.500560:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826344.500562:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826344.500565:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826344.516388:0:22705:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826344.516424:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346288512 00000100:00100000:3.0:1787826344.516434:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346288512 00000100:00100000:3.0:1787826344.516495:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43641 00000100:00100000:3.0:1787826344.516506:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c5063050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346288512:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826344.516523:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e87608018bd lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4bad1 expref: -99 pid: 225995 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826344.516630:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826344.516642:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c5063050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346288512:12345-192.168.203.156@tcp:104: Request processed in 135us (2736us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826344.516655:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43641 00020000:00200000:3.0:1787826344.516734:0:225995:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:3.0:1787826344.516739:0:225995:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826344.516758:0:225995:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:3.0:1787826344.516771:0:225995:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826344.516777:0:225995:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a714b0, osc lock ffff9057c50ba850, flags 0x0 00010000:00010000:3.0:1787826344.516832:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.516855:0:225995:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.516880:0:225995:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.516902:0:225995:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332650752:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826344.516953:0:225995:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000008:00010000:0.0:1787826344.542667:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000005746e05d ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826344.542731:0:225995:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332650752:192.168.203.156@tcp:101:cat.0 00000008:00010000:0.0:1787826344.542746:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000005746e05d ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b86cd977/0x95194e8760801885 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0xe21309aa81f4b9b2 expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826344.542761:0:225995:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.542775:0:225995:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826344.542778:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332651008:192.168.203.156@tcp:4:bash.0 00010000:00010000:3.0:1787826344.542797:0:225995:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826344.547554:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346289024 00000100:00100000:3.0:1787826344.547568:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346289024 00000100:00100000:3.0:1787826344.547706:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43643 00000100:00100000:3.0:1787826344.547720:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:125406:x1874663346289024:12345-192.168.203.156@tcp:105: 00010000:00010000:3.0:1787826344.547741:0:6158:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547756:0:6158:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547767:0:6158:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547880:0:6158:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547893:0:6158:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547911:0:6158:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x80000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826344.547951:0:6158:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9058021f9200) 00000100:00100000:3.0:1787826344.548314:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:125406:x1874663346289024:12345-192.168.203.156@tcp:105: Request processed in 596us (1092us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826344.548323:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43643 00000020:00010000:0.0:1787826344.576988:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:0.0:1787826344.577003:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826344.577008:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057edd1e9b0 (ffff9057d889c2d8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826344.577022:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826344.577042:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826344.577063:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826344.577067:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057edd1e9b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826344.577082:0:225995:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826344.577098:0:225995:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826344.577104:0:225995:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826344.577109:0:225995:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:0.0:1787826344.577138:0:225995:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dbf:0x0], ppos: 6, count: 4194304 00000080:00200000:0.0:1787826344.577151:0:225995:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:0.0:1787826344.577156:0:225995:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826344.577162:0:225995:0:(cl_io.c:214:cl_io_rw_init()) header@000000002d7549b3[0x110, 2, [0x200000403:0x1dbf:0x0] hash] 00000020:00200000:0.0:1787826344.577164:0:225995:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:0.0:1787826344.577168:0:225995:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbf:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826344.577187:0:225995:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:0.0:1787826344.577190:0:225995:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826344.577208:0:225995:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:0.0:1787826344.577248:0:225995:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826344.577255:0:225995:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d5b0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826344.577286:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577303:0:225995:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577316:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577326:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577807:0:225995:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826344.577819:0:225995:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:0.0:1787826344.577840:0:225995:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dbf:0x0] 00000008:00010000:0.0:1787826344.577848:0:225995:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826344.577851:0:225995:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d830, osc lock ffff9057c8b838f0, flags 0x400001002 00010000:00010000:0.0:1787826344.577862:0:225995:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577872:0:225995:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826344.577882:0:225995:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826344.577886:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577893:0:225995:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826344.577900:0:225995:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826344.577908:0:225995:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dbf:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826344.577911:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:0.0:1787826344.577914:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826344.577917:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d830 (ffff9057d889c2d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826344.577920:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826344.577922:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d830 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826344.577930:0:225995:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:0.0:1787826344.577938:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:0.0:1787826344.577940:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826344.577943:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d5b0 (ffff9057d889c2d8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826344.577947:0:225995:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826344.577962:0:225995:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:0.0:1787826344.577972:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826344.577975:0:225995:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d5b0 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826344.577980:0:225995:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826344.577986:0:225995:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbf:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826344.577990:0:225995:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:0.0:1787826344.577993:0:225995:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:0.0:1787826344.578120:0:225995:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dbf:0x0](ffff9057dfe0ba88), flags 100000 00000002:00100000:0.0:1787826344.578203:0:225995:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158c700 x1874663332648064/t8589957449(8589957449) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826359 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826344.578217:0:225995:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826344.578287:0:225995:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f480 x1874663332651648/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826344.578314:0:225995:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332651648:192.168.203.156@tcp:35:cat.0 00000100:00100000:0.0:1787826344.581156:0:225995:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826344.596293:0:225995:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:225995:1874663332651648:192.168.203.156@tcp:35:cat.0 00000002:00100000:0.0:1787826344.596348:0:225995:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158c700 x1874663332648064/t8589957449(8589957449) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826359 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826344.599929:0:225997:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:225997:1874663332650240:192.168.203.156@tcp:36:mrename.0 00000100:00100000:3.0:1787826345.164549:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346292864 00000100:00100000:3.0:1787826345.164564:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346292864 00000100:00100000:3.0:1787826345.164643:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43644 00000100:00100000:3.0:1787826345.164654:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:20167:x1874663346292864:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826345.164677:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826345.164802:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826345.164810:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:20167:x1874663346292864:12345-192.168.203.156@tcp:104: Request processed in 158us (1088us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826345.164817:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43644 00010000:00010000:0.0:1787826345.165371:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826345.165380:0:15672:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.165399:0:15672:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9200 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826345.165408:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800428400030000 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826345.165432:0:15672:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826345.165508:0:15672:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826345.165563:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.165571:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826345.165610:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.165619:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000046ab98ac/0x95194e87608018ee lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x522:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4bb09 expref: -99 pid: 225995 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.165646:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826345.165648:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826345.165650:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826345.381301:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332651904:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826345.381440:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826345.381449:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332652032:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826345.389912:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332652032:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826345.398054:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826345.398111:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000001c6df59e/0x95194e8760801893 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4b9dc expref: -99 pid: 225814 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826345.398153:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332651904:192.168.203.156@tcp:4:bash.0 00000080:00200000:0.0:1787826345.747207:0:226033:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 4096 00000002:00100000:0.0:1787826345.747270:0:226033:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb238000 x1874663332652672/t4294985451(4294985451) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826345.747369:0:226033:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e56:0x0](ffff9057dfe0b248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826345.747378:0:226033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248),name=d45j.sanityn 00000002:00010000:0.0:1787826345.747407:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e56:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.747424:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747438:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826345.747450:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.747456:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747465:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747474:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747482:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826345.747498:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.747507:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747515:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747522:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747534:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826345.747544:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826345.747550:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) 00000080:00010000:0.0:1787826345.747554:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) for lock 0x95194e87608018fc 00000080:00010000:0.0:1787826345.747559:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018fc from it ffffb512c8093cd8 00010000:00010000:0.0:1787826345.747562:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.747570:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826345.747581:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.747588:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747596:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747603:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747610:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826345.747617:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.747622:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747630:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747637:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.747643:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826345.747741:0:226033:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826345.752765:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826345.752844:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826345.752858:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.752868:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826345.753726:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332652800:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826345.753874:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.764669:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332652800:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826345.767334:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.767361:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826345.767393:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826345.767413:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.767428:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826345.768676:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332652928:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826345.770285:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.782736:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332652928:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826345.783540:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.783560:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826345.783727:0:226033:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e56:0x0](ffff9057dfe0b248), flags 704000 00000002:00100000:0.0:1787826345.783812:0:226033:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb238000 x1874663332652672/t4294985451(4294985451) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826345.783825:0:226033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826345.783831:0:226033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332653184/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826345.783866:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653184:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826345.783988:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.790621:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653184:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826345.790669:0:226033:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb238000 x1874663332652672/t4294985451(4294985451) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826345.790881:0:226033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826345.790905:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.790927:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.791083:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826345.791095:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826345.791102:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826345.791105:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e876080189a 00000080:00010000:0.0:1787826345.791114:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080189a from it ffffb512c8093bf8 00010000:00010000:0.0:1787826345.791119:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.791130:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826345.791147:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826345.791157:0:226033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826345.791166:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), inode mode 41ed mask 44 00000080:00200008:0.0:1787826345.791176:0:226033:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e56:0x0](ffff9057dfe0b248), flags 704400 00000002:00010000:0.0:1787826345.791194:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e56:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826345.791452:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.791462:0:226033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826345.791476:0:226033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826345.791485:0:226033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332653440/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826345.791495:0:226033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826345.791519:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653440:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826345.792392:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.801939:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653440:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826345.802007:0:226033:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.802020:0:226033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.802033:0:226033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.802045:0:226033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826345.802064:0:226033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158c700 x1874663332653440/t4294985453(4294985453) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826361 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826345.802078:0:226033:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158c700 x1874663332653440/t4294985453(4294985453) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826361 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826345.802094:0:226033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.802106:0:226033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7766 (0 0) 00000080:00200000:0.0:1787826345.802114:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826345.802155:0:226033:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e56:0x0], updating i_size 4096 00000080:00010000:0.0:1787826345.802164:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) for lock 0x95194e8760801918 00000080:00010000:0.0:1787826345.802173:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801918 from it ffffb512c8093bd0 00010000:00010000:0.0:1787826345.802178:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.802190:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826345.802282:0:226033:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158c700 x1874663332653440/t4294985453(4294985453) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826345.802380:0:226033:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e56:0x0](ffff9057dfe0b248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826345.802388:0:226033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248),name=d45j.sanityn 00000002:00010000:0.0:1787826345.802412:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e56:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.802425:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.802440:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826345.802451:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.802457:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802465:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802475:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802484:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826345.802492:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.802500:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802509:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802517:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826345.802524:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826345.802535:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826345.802541:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) 00000080:00010000:0.0:1787826345.802545:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e56:0x0](ffff9057dfe0b248) for lock 0x95194e87608018fc 00000080:00010000:0.0:1787826345.802549:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608018fc from it ffffb512c8093cd8 00010000:00010000:0.0:1787826345.802552:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.802559:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826345.802571:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.802577:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802585:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802593:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802600:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826345.802608:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.802613:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802626:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802633:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826345.802639:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826345.802731:0:226033:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057d128a800) d45j.sanityn proj=0 00000080:00000004:0.0:1787826345.802736:0:226033:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057d128a800: age 8155, max_age 8160 00800000:00000004:0.0:1787826345.802748:0:226033:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: age 8155, max_age 8160 00000100:00100000:0.0:1787826345.804062:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653568:192.168.203.156@tcp:41:rm.0 00000100:00100000:0.0:1787826345.804187:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.807650:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653568:192.168.203.156@tcp:41:rm.0 00800000:00000004:0.0:1787826345.807687:0:226033:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793371/1916558 objects 519509/534204 00000080:00000004:0.0:1787826345.807695:0:226033:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793371/1916558 objects 519509/534204 00000080:00000004:0.0:1787826345.807701:0:226033:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793371/1916558 objects 519509/534204 00000080:00200000:0.0:1787826345.807755:0:226033:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826345.807804:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826345.807859:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x199:0x0]) in obj [0x240000400:0x199:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826345.807874:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.807891:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826345.807906:0:226033:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826345.807911:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.807921:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e8760801903 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb4f expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:0.0:1787826345.807939:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x197:0x0]) in obj [0x200000401:0x197:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826345.807960:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.807969:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826345.807980:0:226033:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826345.807983:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.807990:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826345.808031:0:226033:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:0.0:1787826345.808060:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787826345.808081:0:226033:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), flags=0 00000080:00200000:0.0:1787826345.808092:0:226033:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), intent=getattr 00000002:00010000:0.0:1787826345.808114:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x197:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826345.808159:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.808168:0:226033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.808178:0:226033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826345.808200:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653696:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826345.808329:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.815506:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653696:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826345.815545:0:226033:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: --/CR res: [0x200000401:0x197:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815570:0:226033:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815580:0:226033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815593:0:226033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815603:0:226033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815623:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815633:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815641:0:226033:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826345.815652:0:226033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d500 x1874663332653696/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826361 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826345.815665:0:226033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815675:0:226033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7767 (0 0) 00000002:00010000:0.0:1787826345.815691:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e57:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826345.815756:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e57:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815764:0:226033:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.815779:0:226033:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226033 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826345.815800:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653824:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826345.815879:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.835197:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332653824:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826345.835283:0:226033:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e57:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835298:0:226033:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835311:0:226033:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835322:0:226033:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835336:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835345:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835353:0:226033:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826345.835367:0:226033:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d880 x1874663332653824/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826361 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826345.835392:0:226033:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835408:0:226033:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7767 (0 0) 00000080:00200000:0.0:1787826345.835432:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x197:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826345.835474:0:226033:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e57:0x0](ffff9057dfe05348) 00000080:00010000:0.0:1787826345.835595:0:226033:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe05348) for remote lock 0x95194e8760801926 00000080:00010000:0.0:1787826345.835602:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe05348) for lock 0x95194e876080191f 00000080:00010000:0.0:1787826345.835613:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e57:0x0](ffff9057dfe05348) 00000080:00010000:0.0:1787826345.835617:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080191f from it ffffb512c8093df8 00010000:00010000:0.0:1787826345.835621:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835633:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826345.835648:0:226033:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801926 from it ffffb512c8093df8 00010000:00010000:0.0:1787826345.835651:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826345.835660:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826345.835683:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248), inode mode 41ed mask 3 00000080:00200000:0.0:1787826345.835692:0:226033:0:(namei.c:2418:ll_unlink()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e56:0x0](ffff9057dfe0b248) 00010000:00010000:0.0:1787826345.835735:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.835796:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.835803:0:226033:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826345.835828:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e876080191f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb9c expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.835855:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.835878:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.835884:0:226033:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826345.835887:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801926 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e57:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bbaa expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826345.835907:0:226033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826345.835914:0:226033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d500 x1874663332654208/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826345.835937:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654208:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826345.836044:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.839011:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332653952:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826345.847081:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346293504 00000100:00100000:3.0:1787826345.847097:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346293504 00000100:00100000:3.0:1787826345.847182:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43645 00000100:00100000:3.0:1787826345.847194:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346293504:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826345.847260:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e876080190a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x197:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bb5d expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826345.847362:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826345.847369:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346293504:12345-192.168.203.156@tcp:104: Request processed in 178us (2166us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826345.847377:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43645 00000100:00100000:0.0:1787826345.870087:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654208:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826345.870163:0:226033:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e57:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826345.870178:0:226033:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826345.870211:0:226033:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e57:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826345.870246:0:226033:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e57:0x0](ffff9057dfe05348) 00000080:00200008:0.0:1787826345.870360:0:226033:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e56:0x0](ffff9057dfe0b248), flags 704000 00010000:00010000:0.0:1787826345.870395:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826345.870441:0:226033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826345.870472:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.870485:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.870498:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826345.870505:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826345.870511:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826345.870532:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e876080189a 00000080:00010000:0.0:1787826345.870540:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080189a from it ffffb512c8093c10 00010000:00010000:0.0:1787826345.870544:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826345.870553:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826345.870570:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826345.870582:0:226033:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=0 00000080:00200000:0.0:1787826345.870589:0:226033:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826345.870595:0:226033:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826345.870601:0:226033:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826345.870609:0:226033:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826345.870621:0:226033:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826345.870625:0:226033:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826345.870627:0:226033:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e876080189a 00000080:00010000:0.0:1787826345.870631:0:226033:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080189a from it ffffb512c8093d88 00010000:00010000:0.0:1787826345.870634:0:226033:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826345.870641:0:226033:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1787826345.870649:0:226033:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826345.870655:0:226033:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988) 00010000:00010000:0.0:1787826345.870677:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608018a1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4ba0d expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.870950:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608018a1 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ba0d expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.870959:0:226033:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826345.870983:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2cc00/0x95194e87608018a1 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ba0d expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.871013:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:0.0:1787826345.871055:0:226033:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158c700 x1874663332653440/t4294985453(4294985453) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826345.871072:0:226033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826345.871077:0:226033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ea00 x1874663332654720/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826345.871097:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654720:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826345.871194:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826345.875019:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654720:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826345.875076:0:226033:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158c700 x1874663332653440/t4294985453(4294985453) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826361 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787826345.875139:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.875158:0:226033:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e87608018a8 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4ba1b expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.875199:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 1/0,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.875206:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.875212:0:226033:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e87608018a8 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ba1b expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826345.875240:0:226033:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787826345.875244:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801918 lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e56:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb79 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826345.875274:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06600/0x95194e87608018fc lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bb41 expref: -99 pid: 226033 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826345.875283:0:226033:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c600/0x95194e87608018a8 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e56:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ba1b expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826345.875306:0:226033:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826345.875313:0:226033:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ea00 x1874663332654976/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826345.875338:0:226033:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654976:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826345.875430:0:226033:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826345.898025:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826345.898127:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826345.898171:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346295040 00000100:00100000:3.0:1787826345.898183:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346295040 00000100:00100000:3.0:1787826345.898305:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346295168 00000100:00100000:3.0:1787826345.898309:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346295168 00000100:00100000:3.0:1787826345.898331:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43646 00000100:00100000:3.0:1787826345.898341:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346295040:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826345.898366:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e87608018c4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ba5a expref: -99 pid: 225997 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826345.898465:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826345.898472:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346295040:12345-192.168.203.156@tcp:104: Request processed in 132us (473us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826345.898479:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43646 00000100:00100000:3.0:1787826345.898498:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43647 00000100:00100000:3.0:1787826345.898504:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb220380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346295168:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826345.898512:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d000/0x95194e876080189a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4b9f1 expref: -99 pid: 225995 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1787826345.898544:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826345.898550:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb220380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346295168:12345-192.168.203.156@tcp:104: Request processed in 46us (439us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826345.898556:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43647 00010000:00010000:0.0:1787826345.899433:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826345.911190:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826345.917106:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332655360:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826345.936198:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826345.937523:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332655360:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826345.960731:0:226033:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226033:1874663332654976:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826345.960796:0:226033:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e56:0x0](ffff9057dfe0b248) 00000080:00200000:3.0:1787826346.014587:0:226041:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057c7539000) / proj=0 00000080:00000004:3.0:1787826346.014602:0:226041:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057c7539000: age 8155, max_age 8160 00800000:00000004:3.0:1787826346.014620:0:226041:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: age 8155, max_age 8160 00000100:00100000:3.0:1787826346.014677:0:226041:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:226041:1874663332655616:192.168.203.156@tcp:41:stat.0 00000100:00100000:3.0:1787826346.014838:0:226041:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826346.017079:0:226041:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957450 gen 1 00000100:00100000:3.0:1787826346.017128:0:226041:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:226041:1874663332655616:192.168.203.156@tcp:41:stat.0 00800000:00000004:3.0:1787826346.017151:0:226041:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: update ffff9057c6ef0450 cache blocks 1793371/1916558 objects 519509/534202 00000080:00000004:3.0:1787826346.017159:0:226041:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057c7539000: update ffff9057c6ef37c0 cache blocks 1793371/1916558 objects 519509/534202 00000080:00000004:3.0:1787826346.017164:0:226041:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793371/1916558 objects 519509/534202 00000080:00200000:3.0:1787826346.077725:0:226045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826346.079462:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826346.079577:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.079591:0:226045:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.079609:0:226045:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826346.079648:0:226045:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332655744:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826346.079918:0:226045:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826346.086817:0:226045:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332655744:192.168.203.156@tcp:101:lfs.0 00010000:00010000:3.0:1787826346.086852:0:226045:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086866:0:226045:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086880:0:226045:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086901:0:226045:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086927:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086937:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086945:0:226045:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826346.086963:0:226045:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221c00 x1874663332655744/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826346.086978:0:226045:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.086997:0:226045:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826346.087193:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826346.087376:0:226045:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826346.087385:0:226045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080192d 00000080:00010000:3.0:1787826346.087392:0:226045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826346.087395:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080192d from it ffffb512c806bb58 00010000:00010000:3.0:1787826346.087406:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.087416:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826346.087458:0:226045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826346.087492:0:226045:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:3.0:1787826346.087498:0:226045:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:3.0:1787826346.087530:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826346.087594:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.087603:0:226045:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.087613:0:226045:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826346.087639:0:226045:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332655872:192.168.203.156@tcp:101:lfs.0 00000100:00100000:3.0:1787826346.087881:0:226045:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826346.115727:0:226045:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332655872:192.168.203.156@tcp:101:lfs.0 00010000:00010000:3.0:1787826346.115757:0:226045:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115781:0:226045:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115788:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115795:0:226045:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115808:0:226045:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115814:0:226045:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826346.115819:0:226045:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115826:0:226045:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115832:0:226045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115839:0:226045:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826346.115846:0:226045:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.115852:0:226045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801934 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826346.115873:0:226045:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057fb221c00 x1874663332655872/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1787826346.115889:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1787826346.115925:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826346.115949:0:226045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1787826346.115958:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080192d from it ffffb512c806b930 00010000:00010000:3.0:1787826346.115963:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826346.115970:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826346.116074:0:226045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826346.116091:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826346.116100:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826346.116433:0:226045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826346.116441:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826346.116448:0:226045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826346.116461:0:226045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080192d 00000080:00010000:3.0:1787826346.116467:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080192d from it ffffb512c806bbe8 00010000:00010000:3.0:1787826346.116471:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826346.116479:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826346.116488:0:226045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826346.116497:0:226045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:3.0:1787826346.116536:0:226045:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826346.116541:0:226045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826346.116550:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826346.116557:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826346.116566:0:226045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826346.116572:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826346.116576:0:226045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826346.116579:0:226045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080192d 00000080:00010000:3.0:1787826346.116582:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080192d from it ffffb512c806bcb8 00010000:00010000:3.0:1787826346.116585:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826346.116592:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1787826346.117763:0:226045:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1787826346.117928:0:226045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826346.118015:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826346.118089:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.118099:0:226045:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.118394:0:226045:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226045 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826346.118430:0:226045:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226045:1874663332656000:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826346.119113:0:226045:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.129987:0:226045:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226045:1874663332656000:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826346.130024:0:226045:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130039:0:226045:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130058:0:226045:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130069:0:226045:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130083:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130093:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130102:0:226045:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.130125:0:226045:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221c00 x1874663332656000/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.130562:0:226045:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130579:0:226045:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826346.130587:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826346.130626:0:226045:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826346.130633:0:226045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e876080193b 00000080:00010000:0.0:1787826346.130640:0:226045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826346.130643:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080193b from it ffffb512c806bcb8 00010000:00010000:0.0:1787826346.130648:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.130657:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9000/0x95194e876080193b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc59 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826346.130989:0:226045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826346.131015:0:226045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826346.131030:0:226045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826346.131043:0:226045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826346.131050:0:226045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826346.131055:0:226045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826346.131058:0:226045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e876080192d 00000080:00010000:0.0:1787826346.131062:0:226045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e876080192d from it ffffb512c806bbb0 00010000:00010000:0.0:1787826346.131064:0:226045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826346.131072:0:226045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826346.131082:0:226045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 44 00000080:00200008:0.0:1787826346.131093:0:226045:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200080:0.0:1787826346.131127:0:226045:0:(dir.c:2001:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) cmd=c03066f0 arg=7ffc3ed2ca60 00000080:00200000:0.0:1787826346.133919:0:226045:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) name=d45j.sanityn stripe_offset=1 stripe_count=2, hash_type=2 00000020:00100000:0.0:1787826346.137484:0:226045:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826346.137888:0:226045:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332656256/t0(0) o36->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826346.137928:0:226045:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332656256:192.168.203.156@tcp:36:lfs.0 00000100:00100000:0.0:1787826346.139444:0:226045:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00010000:00010000:0.0:1787826346.161215:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826346.161262:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161277:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f8e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826346.161281:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161289:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161296:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161339:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161345:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826346.161351:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161357:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8e00/0x95194e876080192d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bc36 expref: -99 pid: 226045 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826346.161375:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826346.161377:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826346.161378:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826346.161403:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332656512:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826346.192501:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332656512:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826346.215821:0:226045:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226045:1874663332656256:192.168.203.156@tcp:36:lfs.0 00000080:00200000:0.0:1787826346.215895:0:226045:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe05348) 00000080:00200000:0.0:1787826346.215904:0:226045:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e58:0x0], updating i_size 4096 00000080:00200008:0.0:1787826346.215945:0:226045:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200000:3.0:1787826346.221473:0:226034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826346.221628:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826346.221718:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.221731:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.221742:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826346.221786:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332656640:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826346.221910:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d9c0 going to sleep for 11 seconds 00000080:00200000:0.0:1787826346.237482:0:226034:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826346.237490:0:226034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801942 00000080:00010000:0.0:1787826346.237495:0:226034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826346.237498:0:226034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801942 from it ffffb512c8083bf8 00010000:00010000:0.0:1787826346.237500:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bc9f expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.237508:0:226034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bc9f expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826346.237618:0:226034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826346.237636:0:226034:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:0.0:1787826346.237640:0:226034:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:0.0:1787826346.237664:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826346.237706:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.237712:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.237720:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826346.237744:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332656768:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826346.237932:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.250126:0:226034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332656768:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826346.250163:0:226034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250191:0:226034:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250201:0:226034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250214:0:226034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250281:0:226034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250298:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250313:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250320:0:226034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.250334:0:226034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158c000 x1874663332656768/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.250348:0:226034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250360:0:226034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7768 (0 0) 00000002:00010000:0.0:1787826346.250380:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e58:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826346.250452:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250461:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.250472:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826346.250497:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332656896:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826346.250608:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.254181:0:226034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332656896:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826346.254215:0:226034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254284:0:226034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254298:0:226034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254311:0:226034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254326:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254335:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254344:0:226034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.254357:0:226034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158ce00 x1874663332656896/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.254373:0:226034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254387:0:226034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7768 (0 0) 00000080:00200000:0.0:1787826346.254410:0:226034:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1787826346.254679:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826346.254869:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254877:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.254885:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826346.254909:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657024:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826346.255023:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.258361:0:226034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657024:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826346.258393:0:226034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258408:0:226034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258431:0:226034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258445:0:226034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258459:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258469:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258482:0:226034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.258495:0:226034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c000 x1874663332657024/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.258509:0:226034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258535:0:226034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/410 (0 0) 00010000:00010000:0.0:1787826346.258545:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258554:0:226034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826346.258587:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826346.258641:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258649:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.258658:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826346.258682:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657152:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826346.258945:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.265999:0:226034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657152:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826346.266128:0:226034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 4/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266143:0:226034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 4/1,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266157:0:226034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266170:0:226034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266185:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266193:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266200:0:226034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.266213:0:226034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c000 x1874663332657152/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826362 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.266269:0:226034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266284:0:226034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/408 (0 0) 00010000:00010000:0.0:1787826346.266295:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266303:0:226034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e876080195e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bce5 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826346.266347:0:226034:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e58:0x0], updating i_size 8192 00000080:00200000:0.0:1787826346.266352:0:226034:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e58:0x0](ffff9057dfe05348): rc = 0 00000080:00010000:0.0:1787826346.266364:0:226034:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe05348) for remote lock 0x95194e8760801950 00000080:00010000:0.0:1787826346.266369:0:226034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe05348) for lock 0x95194e8760801949 00000080:00010000:0.0:1787826346.266383:0:226034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe05348) 00000080:00010000:0.0:1787826346.266386:0:226034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801949 from it ffffb512c8083ac8 00010000:00010000:0.0:1787826346.266390:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266399:0:226034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826346.266407:0:226034:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801950 from it ffffb512c8083ac8 00010000:00010000:0.0:1787826346.266411:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266431:0:226034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801950 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bcc9 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826346.266457:0:226034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe05348), inode mode 41ed mask 1 00000080:00200000:0.0:1787826346.266485:0:226034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe05348), inode mode 41ed mask 3 00000080:00200000:0.0:1787826346.266491:0:226034:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e58:0x0](ffff9057dfe05348), file ffff9057c6024c00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826346.266502:0:226034:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e58:0x0](ffff9057dfe05348), intent=open|creat 00000002:00010000:0.0:1787826346.266543:0:226034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x240000403:0x1e59:0x0]) in obj [0x240000400:0x19a:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826346.266559:0:226034:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826346.266621:0:226034:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826346.266629:0:226034:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826346.266631:0:226034:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801957 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bcd7 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826346.266675:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.266683:0:226034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826346.266908:0:226034:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826346.266920:0:226034:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ce00 x1874663332657408/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826346.266931:0:226034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826346.267074:0:226034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657408:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826346.267353:0:226034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.274570:0:226034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226034:1874663332657408:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826346.274607:0:226034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/0,1 mode: --/CW res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274645:0:226034:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274655:0:226034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274669:0:226034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274680:0:226034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274826:0:226034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274837:0:226034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826346.274844:0:226034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826346.274855:0:226034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158ce00 x1874663332657408/t4294985458(4294985458) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826362 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826346.274865:0:226034:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ce00 x1874663332657408/t4294985458(4294985458) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826362 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826346.274876:0:226034:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801965 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd01 expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826346.336456:0:226049:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826346.336498:0:226049:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787826346.336673:0:226048:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:0.0:1787826346.336692:0:226048:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1787826346.336742:0:226050:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:0.0:1787826346.340412:0:226050:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/7 locks to regular blp list, 3 blwis in pool 00010000:00010000:3.0:1787826346.343176:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 4 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1787826346.343197:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07a00/0x95194e8760801973 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4bd2b expref: -99 pid: 226034 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826346.343260:0:15672:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc0:0x0]: losing layout lock 00010000:00010000:3.0:1787826346.343311:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bc9f expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826346.343328:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826346.343382:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07a00/0x95194e8760801973 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4bd2b expref: -99 pid: 226034 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826346.343395:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bc9f expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826346.343402:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826346.343408:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:3.0:1787826346.343443:0:15672:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:fc0b407b-a541-420d-8712-b9c8da18fe8c:15672:1874663332658560:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826346.343636:0:15672:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826346.347849:0:15672:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:fc0b407b-a541-420d-8712-b9c8da18fe8c:15672:1874663332658560:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826346.347890:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07a00/0x95194e8760801973 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4bd2b expref: -99 pid: 226034 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826346.347924:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801942 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bc9f expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826346.347941:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3400/0x95194e8760801949 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bcbb expref: -99 pid: 226034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826346.347984:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826346.347991:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2ca00/0x95194e87608018f5 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4bb1e expref: -99 pid: 225995 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826346.348023:0:15672:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dbf:0x0]: losing layout lock 00010000:00010000:0.0:1787826346.348087:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2ca00/0x95194e87608018f5 lrc: 1/0,0 mode: --/CR res: [0x200000403:0x1dbf:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4bb1e expref: -99 pid: 225995 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826346.348096:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826346.348118:0:15672:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:398e0287-d3c7-44dd-b1d9-6b85f698e615:15672:1874663332658688:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826346.348258:0:15672:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51c00 going to sleep for 11 seconds 00010000:00010000:3.0:1787826346.791177:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826347.715002:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826347.810492:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826349.475919:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332659200:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476064:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332659456:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476094:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332659712:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476118:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332659968:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476149:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 5 async RPCs [0->1] 00000100:00100000:3.0:1787826349.476154:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332659072:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476184:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332659328:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476489:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332659584:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476514:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332659840:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826349.476536:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332660096:192.168.203.156@tcp:400:kworker.0 00000080:00200000:0.0:1787826350.501617:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e57:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826350.501640:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826350.501674:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e57:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826350.501690:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826350.501695:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787826350.501719:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000044224c7a ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826350.501742:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000044224c7a ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826350.501817:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1787826350.501920:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158e300 - object 0x2c0000400:1229 offset 0<>6 00000100:00100000:0.0:1787826350.502018:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332660352:192.168.203.156@tcp:16:kworker.0 00000100:00100000:0.0:1787826350.502152:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826350.502159:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332660224:192.168.203.156@tcp:4:bash.0 00000100:00100000:0.0:1787826350.507196:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332660352:192.168.203.156@tcp:16:kworker.0 00000080:00200000:0.0:1787826350.527147:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826351.010761:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826351.010786:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826351.010821:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826351.452404:0:226215:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826351.452441:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826351.452514:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.452525:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.452536:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826351.452566:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660736:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826351.452699:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826351.464858:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660736:192.168.203.156@tcp:101:cat.0 00010000:00010000:3.0:1787826351.464895:0:226215:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464910:0:226215:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464925:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464938:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464962:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464972:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.464980:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826351.464993:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158e680 x1874663332660736/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826351.465006:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.465025:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826351.465033:0:226215:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826351.465065:0:226215:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826351.465072:0:226215:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:3.0:1787826351.465078:0:226215:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826351.465082:0:226215:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c824bbf8 00010000:00010000:3.0:1787826351.465086:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.465096:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826351.465127:0:226215:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826351.465145:0:226215:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=25 00000080:00200000:3.0:1787826351.465150:0:226215:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=0 00000002:00010000:3.0:1787826351.465176:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826351.465250:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.465259:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826351.465267:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826351.465290:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660864:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826351.465527:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826351.475207:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660864:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826351.475269:0:226215:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475296:0:226215:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475307:0:226215:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475320:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475332:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475346:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475360:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475367:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826351.475379:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb222300 x1874663332660864/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826351.475392:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475403:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7768 (0 0) 00000002:00010000:0.0:1787826351.475424:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e58:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826351.475483:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475492:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.475502:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826351.475528:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660992:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826351.475660:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826351.486706:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332660992:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826351.486747:0:226215:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486762:0:226215:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486775:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486787:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486801:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486809:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486816:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826351.486833:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332660992/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826351.486846:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.486860:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7768 (0 0) 00000080:00200000:0.0:1787826351.486883:0:226215:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826351.486928:0:226215:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148) 00000002:00010000:0.0:1787826351.487079:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826351.487138:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.487148:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.487158:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826351.487182:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332661120:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826351.487305:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826351.500331:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332661120:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826351.500367:0:226215:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500381:0:226215:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500395:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500406:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500421:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500430:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500439:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826351.500451:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f100 x1874663332661120/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826351.500464:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500478:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/410 (0 0) 00010000:00010000:0.0:1787826351.500487:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500496:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826351.500530:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826351.500587:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500594:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.500603:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826351.500639:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332661248:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826351.500938:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826351.506572:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332661248:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826351.506617:0:226215:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 4/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506632:0:226215:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506646:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506657:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506671:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506680:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506688:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826351.506701:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f100 x1874663332661248/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826351.506715:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506730:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/408 (0 0) 00010000:00010000:0.0:1787826351.506741:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.506750:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e876080199d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd94 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826351.507519:0:226215:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e58:0x0], updating i_size 8192 00000080:00010000:0.0:1787826351.507537:0:226215:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) for remote lock 0x95194e876080198f 00000080:00010000:0.0:1787826351.507546:0:226215:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) for lock 0x95194e8760801988 00000080:00010000:0.0:1787826351.507561:0:226215:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) 00000080:00010000:0.0:1787826351.507566:0:226215:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801988 from it ffffb512c824bac8 00010000:00010000:0.0:1787826351.507571:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.507581:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826351.507591:0:226215:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e876080198f from it ffffb512c824bac8 00010000:00010000:0.0:1787826351.507595:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826351.507603:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826351.907506:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826352.004685:0:226217:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826352.004712:0:226217:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826352.004773:0:226217:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226217 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.004785:0:226217:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226217 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.004795:0:226217:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226217 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826352.004819:0:226217:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226217:1874663332661632:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826352.004918:0:226217:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5df00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826352.011345:0:226217:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957457 gen 1 00000100:00100000:3.0:1787826352.011394:0:226217:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226217:1874663332661632:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826352.011423:0:226217:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.011438:0:226217:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.011452:0:226217:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.011463:0:226217:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.012415:0:226217:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.012441:0:226217:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.012448:0:226217:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826352.012464:0:226217:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222300 x1874663332661632/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826368 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826352.012489:0:226217:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.012505:0:226217:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826352.012513:0:226217:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:0.0:1787826352.067010:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332662272:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826352.084366:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346305664 00000100:00100000:3.0:1787826352.084378:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346305664 00000100:00100000:3.0:1787826352.084443:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43653 00000100:00100000:3.0:1787826352.084455:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346305664:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826352.084478:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084565:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826352.084572:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346305664:12345-192.168.203.156@tcp:104: Request processed in 120us (461us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826352.084579:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43653 00010000:00010000:3.0:1787826352.084603:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826352.084609:0:15672:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084619:0:15672:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826352.084624:0:15672:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084633:0:15672:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084642:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084711:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084720:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826352.084740:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084754:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801996 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bd86 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.084770:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826352.084772:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826352.084773:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826352.085550:0:226215:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e59:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826352.085559:0:226215:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e59:0x0](ffff9057dfe080c8) for lock 0x95194e87608019a4 00000080:00010000:3.0:1787826352.085571:0:226215:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e59:0x0](ffff9057dfe080c8) 00000080:00200008:3.0:1787826352.085589:0:226215:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x240000403:0x1e59:0x0](ffff9057dfe080c8), flags 100000 00000002:00100000:3.0:1787826352.085616:0:226215:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057d0de1880 x1874663332661504/t4294985461(4294985461) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826367 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787826352.085682:0:226215:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019a4 from it ffff9057f161c2a0 00010000:00010000:3.0:1787826352.085688:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.085699:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826352.085722:0:226215:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e59:0x0](ffff9057dfe080c8), inode mode 81a4 mask 44 00000080:00200008:3.0:1787826352.085763:0:226215:0:(file.c:6296:ll_getattr_dentry()) START file f45j.sanityn:[0x240000403:0x1e59:0x0](ffff9057dfe080c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826352.085771:0:226215:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e59:0x0](ffff9057dfe080c8),name=f45j.sanityn 00000002:00010000:3.0:1787826352.085794:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e59:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826352.085804:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.085815:0:226215:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826352.085822:0:226215:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x19a:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826352.085828:0:226215:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e59:0x0](ffff9057dfe080c8) 00000080:00010000:3.0:1787826352.085831:0:226215:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e59:0x0](ffff9057dfe080c8) for lock 0x95194e87608019a4 00000080:00010000:3.0:1787826352.085835:0:226215:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019a4 from it ffffb512c824bcd8 00010000:00010000:3.0:1787826352.085838:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.085850:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826352.085869:0:226215:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1787826352.085879:0:226215:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240000403:0x1e59:0x0] 00000008:00010000:3.0:1787826352.085899:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826352.085903:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71bb0, osc lock ffff9057c50ba4c0, flags 0x400001002 00010000:00010000:3.0:1787826352.085945:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000742a81dc/0x95194e87608019ce lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826352.085956:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000742a81dc/0x95194e87608019ce lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787826352.085977:0:226215:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826352.088503:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332662272:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826352.091775:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826352.091814:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826352.091858:0:22705:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826352.091880:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346305792 00000100:00100000:3.0:1787826352.091890:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346305792 00000100:00100000:3.0:1787826352.091964:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43654 00000100:00100000:3.0:1787826352.091978:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35096:x1874663346305792:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826352.091999:0:22705:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.092015:0:22705:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826352.092053:0:22705:0:(cl_object.c:322:cl_object_glimpse()) header@000000000a33659d[0x110, 2, [0x240000403:0x1e59:0x0] hash] 00000020:00010000:3.0:1787826352.092055:0:22705:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826346 atime: 1787826346 ctime: 1787826346 blocks: 8 00000020:00010000:3.0:1787826352.092059:0:22705:0:(cl_object.c:322:cl_object_glimpse()) header@00000000629914e8[0x110, 2, [0x2c0000400:0x4cd:0x0] hash] 00000020:00010000:3.0:1787826352.092061:0:22705:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826346 atime: 1787826346 ctime: 1787826346 blocks: 8 00000100:00100000:3.0:1787826352.092106:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35096:x1874663346305792:12345-192.168.203.156@tcp:106: Request processed in 131us (345us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826352.092430:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43654 00000100:00100000:0.0:1787826352.092459:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332662912:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1787826352.096120:0:226215:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e59:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826352.096128:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826352.096131:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826352.096137:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71bb0 (ffff9057d889c5a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826352.096142:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826352.096145:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71bb0 (ffff9057f989ccf8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826352.096160:0:226215:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787826352.096341:0:226215:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x240000403:0x1e59:0x0], ppos: 0, count: 4194304 00000080:00200000:0.0:1787826352.096368:0:226215:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:0.0:1787826352.096388:0:226215:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:0.0:1787826352.096394:0:226215:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826352.096401:0:226215:0:(cl_io.c:214:cl_io_rw_init()) header@00000000adaad18b[0x110, 1, [0x240000403:0x1e59:0x0] hash] 00000020:00200000:0.0:1787826352.096404:0:226215:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826352.096409:0:226215:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826352.096428:0:226215:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826352.096432:0:226215:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826352.096450:0:226215:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826352.096465:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826352.096470:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5db30, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826352.096529:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.096539:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.096550:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826352.096578:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663040:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826352.096670:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826352.101809:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332662912:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826352.107921:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346307200 00000100:00100000:3.0:1787826352.107935:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346307200 00000100:00100000:3.0:1787826352.108006:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43656 00000100:00100000:3.0:1787826352.108017:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346307200:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826352.108038:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06c00/0x95194e87608019a4 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4be27 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.108140:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826352.108148:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346307200:12345-192.168.203.156@tcp:104: Request processed in 133us (810us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826352.108156:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43656 00000100:00100000:3.0:1787826352.108816:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826352.108843:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346307328 00000100:00100000:3.0:1787826352.108849:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346307328 00000100:00100000:3.0:1787826352.108877:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43657 00000100:00100000:3.0:1787826352.108885:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:17826:x1874663346307328:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826352.108898:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e876080196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4bd0f expref: -99 pid: 226034 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.109444:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826352.109455:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:17826:x1874663346307328:12345-192.168.203.156@tcp:104: Request processed in 569us (640us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826352.109463:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43657 00010000:00010000:0.0:1787826352.115792:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826352.123880:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826352.124006:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663040:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826352.124044:0:226215:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.124062:0:226215:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.124073:0:226215:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125006:0:226215:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125017:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125033:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826352.125048:0:226215:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125061:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125070:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826352.125102:0:226215:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000020:00010000:0.0:1787826352.125117:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826352.125121:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826352.125125:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5db30 (ffff9057d889c5a8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826352.125130:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125140:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826352.125157:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826352.125161:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5db30 (ffff9057f989ccf8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826352.125169:0:226215:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826352.125182:0:226215:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826352.125186:0:226215:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 1 00000080:00200000:0.0:1787826352.125190:0:226215:0:(file.c:2245:ll_file_io_generic()) f45j.sanityn: restart read from ppos=0 bytes=4194304 retries=999 ret=0: rc = 0 00000080:00200000:0.0:1787826352.125196:0:226215:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826352.125202:0:226215:0:(cl_io.c:214:cl_io_rw_init()) header@00000000adaad18b[0x110, 1, [0x240000403:0x1e59:0x0] hash] 00000020:00200000:0.0:1787826352.125205:0:226215:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826352.125246:0:226215:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826352.125280:0:226215:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x240000403:0x1e59:0x0](ffff9057dfe080c8) 00000002:00010000:0.0:1787826352.125292:0:226215:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e59:0x0]) in obj [0x240000403:0x1e59:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826352.125337:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125346:0:226215:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.125357:0:226215:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226215 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826352.125377:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663424:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826352.125455:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826352.140739:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663424:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826352.140771:0:226215:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.140790:0:226215:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826352.140807:0:226215:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158c380 x1874663332663424/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826368 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826352.140819:0:226215:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826352.140833:0:226215:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.140853:0:226215:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7769 (0 0) 00000080:00200000:0.0:1787826352.140861:0:226215:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x240000403:0x1e59:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826352.140875:0:226215:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e59:0x0](ffff9057dfe080c8) for lock 0x95194e87608019dc 00000080:00010000:0.0:1787826352.140883:0:226215:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019dc from it ffffb512c824baf8 00010000:00010000:0.0:1787826352.140888:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826352.140900:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.140931:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 2/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.140943:0:226215:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 2/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826352.140953:0:226215:0:(file.c:6948:ll_layout_lock_set()) ### file [0x240000403:0x1e59:0x0](0000000065ea5d44) being reconfigured ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826352.140999:0:226215:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e59:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826352.141005:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.141014:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826352.141044:0:226215:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826352.141048:0:226215:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826352.141069:0:226215:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826352.141086:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826352.141092:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5dfb0, osc lock ffff9057c8b82390, flags 0x0 00010000:00010000:0.0:1787826352.141110:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.141134:0:226215:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.141148:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.141156:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826352.141163:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826352.141175:0:226215:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826352.141195:0:226215:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240000403:0x1e59:0x0] 00000008:00010000:0.0:1787826352.141200:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826352.141202:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d0b0, osc lock ffff9057c8b838f0, flags 0x400001002 00010000:00010000:0.0:1787826352.142670:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:3.0:1787826352.150634:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826352.150648:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826352.150654:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5dfb0 (ffff9057d889c5a8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826352.150666:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826352.150796:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826352.150820:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826352.150824:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5dfb0 (ffff9057f989ccf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826352.150841:0:226215:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:3.0:1787826352.150857:0:226215:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826352.150862:0:226215:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:3.0:1787826352.150868:0:226215:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:3.0:1787826352.150895:0:226215:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x240000403:0x1e59:0x0], ppos: 6, count: 4194304 00000080:00200000:3.0:1787826352.150908:0:226215:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:3.0:1787826352.150913:0:226215:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826352.150920:0:226215:0:(cl_io.c:214:cl_io_rw_init()) header@00000000adaad18b[0x110, 2, [0x240000403:0x1e59:0x0] hash] 00000020:00200000:3.0:1787826352.150922:0:226215:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:3.0:1787826352.150926:0:226215:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826352.150946:0:226215:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:3.0:1787826352.150951:0:226215:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826352.150969:0:226215:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:3.0:1787826352.150989:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826352.150995:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a71930, osc lock ffff9057c50bb560, flags 0x0 00010000:00010000:3.0:1787826352.151016:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151032:0:226215:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151046:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151054:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151062:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826352.151138:0:226215:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:3.0:1787826352.151157:0:226215:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240000403:0x1e59:0x0] 00000008:00010000:3.0:1787826352.151164:0:226215:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1787826352.151166:0:226215:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a717b0, osc lock ffff9057c50ba5f0, flags 0x400001002 00010000:00010000:3.0:1787826352.151174:0:226215:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151183:0:226215:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826352.151197:0:226215:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1787826352.151201:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 7/2,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151217:0:226215:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 6/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1787826352.151251:0:226215:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1787826352.151258:0:226215:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e59:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826352.151260:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826352.151263:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826352.151265:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a717b0 (ffff9057d889c5a8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826352.151267:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057e166cde0 (ffff9057f989ccf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826352.151270:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a717b0 (ffff9057f989ccf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826352.151276:0:226215:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:3.0:1787826352.151281:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_release():245 00000020:00010000:3.0:1787826352.151283:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826352.151285:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a71930 (ffff9057d889c5a8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1787826352.151288:0:226215:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826352.151296:0:226215:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00010000:3.0:1787826352.151304:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2d41f38 (ffff9057f989ccf8/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826352.151307:0:226215:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a71930 (ffff9057f989ccf8/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826352.151310:0:226215:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1787826352.151315:0:226215:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826352.151317:0:226215:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1787826352.151319:0:226215:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1787826352.151420:0:226215:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x240000403:0x1e59:0x0](ffff9057dfe080c8), flags 100000 00000002:00100000:3.0:1787826352.151554:0:226215:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057d0de1880 x1874663332661504/t4294985461(4294985461) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826367 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826352.151691:0:226215:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826352.151699:0:226215:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb222d80 x1874663332663808/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826352.151725:0:226215:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663808:192.168.203.156@tcp:35:cat.0 00000100:00100000:3.0:1787826352.151907:0:226215:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d780 going to sleep for 11 seconds 00000100:00100000:3.0:1787826352.155328:0:226215:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226215:1874663332663808:192.168.203.156@tcp:35:cat.0 00000002:00100000:3.0:1787826352.155379:0:226215:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057d0de1880 x1874663332661504/t4294985461(4294985461) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826367 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826352.158015:0:226217:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226217:1874663332662528:192.168.203.156@tcp:36:mrename.0 00000080:00200000:3.0:1787826353.238701:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826353.238752:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826353.238779:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826353.238799:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826353.238808:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826353.238818:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826353.238823:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:3.0:1787826353.238833:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c82bbb58 00010000:00010000:3.0:1787826353.238838:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826353.238847:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826353.238867:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826353.238876:0:226253:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:3.0:1787826353.239546:0:226253:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1787826353.239557:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148),name=d45j.sanityn 00000002:00010000:3.0:1787826353.239580:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e58:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826353.239592:0:226253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7768 (0 0) 00010000:00010000:3.0:1787826353.239654:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826353.239662:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826353.239673:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826353.239706:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332663936:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826353.239909:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000002:00010000:0.0:1787826353.251159:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.253441:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.253461:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.253475:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.253512:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664192:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826353.253897:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.259433:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664192:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826353.259467:0:226253:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x198:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259482:0:226253:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259496:0:226253:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259508:0:226253:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259523:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259532:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259552:0:226253:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.259573:0:226253:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332664192/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.259593:0:226253:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259608:0:226253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/408 (0 0) 00010000:00010000:0.0:1787826353.259618:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259627:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826353.259759:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826353.259785:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.259797:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.259808:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.259818:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.259824:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826353.259827:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:0.0:1787826353.259832:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c82bbbf8 00010000:00010000:0.0:1787826353.259834:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.259840:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826353.259845:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826353.259852:0:226253:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826353.259859:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148), inode mode 41ed mask 44 00000080:00200008:0.0:1787826353.259867:0:226253:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), flags 704400 00000002:00010000:0.0:1787826353.259881:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e58:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1787826353.259937:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.259946:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826353.259958:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826353.259966:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c380 x1874663332664448/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826353.259976:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.260005:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664448:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826353.260294:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.265920:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664448:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826353.265951:0:226253:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.265969:0:226253:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.265978:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.265985:0:226253:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.265996:0:226253:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.266003:0:226253:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826353.266008:0:226253:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.266016:0:226253:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.266023:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.266031:0:226253:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826353.266040:0:226253:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.266048:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e87608019f8 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.266080:0:226253:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826353.266093:0:226253:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826353.266111:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826353.266152:0:226253:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e58:0x0], updating i_size 4096 00000002:00100000:0.0:1787826353.266176:0:226253:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826353.266460:0:226253:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826353.266470:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148),name=d45j.sanityn 00000002:00010000:0.0:1787826353.266497:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e58:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.266523:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266538:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826353.266549:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.266557:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266565:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266576:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266586:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826353.266601:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.266610:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266619:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266627:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266635:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.266645:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826353.266651:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) 00000080:00010000:0.0:1787826353.266655:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) for lock 0x95194e87608019e3 00000080:00010000:0.0:1787826353.266659:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019e3 from it ffffb512c82bbcd8 00010000:00010000:0.0:1787826353.266663:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.266670:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826353.266683:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.266689:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266697:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266704:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266711:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826353.266718:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.266722:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266729:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266737:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.266745:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826353.266843:0:226253:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826353.266859:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826353.266884:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826353.266891:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.266900:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826353.267599:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664576:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826353.267735:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826353.275937:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346309248 00000100:00100000:3.0:1787826353.275960:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346309248 00000100:00100000:3.0:1787826353.276052:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43659 00000100:00100000:3.0:1787826353.276065:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346309248:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.276088:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826353.276276:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826353.276284:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:35352:x1874663346309248:12345-192.168.203.156@tcp:104: Request processed in 222us (2634us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.276291:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43659 00010000:00010000:0.0:1787826353.276346:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826353.276364:0:15673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.276376:0:15673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3600 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826353.276384:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800428400030000 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826353.276409:0:15673:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826353.276520:0:15673:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826353.277734:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.277752:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.277799:0:15673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.277810:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a77df464/0x95194e87608019d5 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x4cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4be74 expref: -99 pid: 226215 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.277843:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.277855:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.277858:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.277869:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826353.277893:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332664704:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.283704:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332664704:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.293139:0:226253:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057d128a800: committing for last_committed 4294985464 gen 1 00000100:00100000:0.0:1787826353.293163:0:226253:0:(client.c:3174:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826353.293180:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664576:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826353.295468:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.295496:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826353.295528:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826353.295547:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.295563:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826353.297827:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664832:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826353.297982:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.311982:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332664832:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826353.312817:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.312839:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826353.312995:0:226253:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), flags 704000 00000002:00100000:0.0:1787826353.313086:0:226253:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826353.313100:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826353.313106:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f100 x1874663332665088/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826353.313132:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665088:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826353.313553:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.337582:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665088:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826353.337641:0:226253:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158c380 x1874663332664448/t4294985464(4294985464) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826353.337750:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826353.337771:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.337794:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.337811:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826353.337820:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.337826:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826353.337830:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:0.0:1787826353.337839:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c82bbbf8 00010000:00010000:0.0:1787826353.337843:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.337854:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826353.337870:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826353.337878:0:226253:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826353.337886:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148), inode mode 41ed mask 44 00000080:00200008:0.0:1787826353.337895:0:226253:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), flags 704400 00000002:00010000:0.0:1787826353.337914:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e58:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826353.337977:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.338206:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826353.338400:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826353.338409:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f100 x1874663332665344/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826353.338425:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.338489:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665344:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826353.338780:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.343821:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665344:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826353.343878:0:226253:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.343893:0:226253:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.343907:0:226253:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.343920:0:226253:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.343938:0:226253:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158f100 x1874663332665344/t4294985466(4294985466) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826353.343952:0:226253:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f100 x1874663332665344/t4294985466(4294985466) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.343967:0:226253:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.343980:0:226253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7768 (0 0) 00000080:00200000:0.0:1787826353.343988:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826353.344029:0:226253:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e58:0x0], updating i_size 4096 00000080:00010000:0.0:1787826353.344036:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) for lock 0x95194e87608019ff 00000080:00010000:0.0:1787826353.344044:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019ff from it ffffb512c82bbbd0 00010000:00010000:0.0:1787826353.344049:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.344060:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826353.344093:0:226253:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f100 x1874663332665344/t4294985466(4294985466) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826353.344191:0:226253:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826353.344212:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148),name=d45j.sanityn 00000002:00010000:0.0:1787826353.344273:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e58:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.344287:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.344302:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826353.344313:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.344320:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344329:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344343:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344351:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826353.344361:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.344370:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344378:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344387:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.344393:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826353.344403:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826353.344409:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) 00000080:00010000:0.0:1787826353.344413:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e58:0x0](ffff9057dfe11148) for lock 0x95194e87608019e3 00000080:00010000:0.0:1787826353.344417:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e87608019e3 from it ffffb512c82bbcd8 00010000:00010000:0.0:1787826353.344420:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.344434:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826353.344446:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.344452:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344460:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344467:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344474:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826353.344482:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.344487:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344494:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344502:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.344509:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826353.344613:0:226253:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057d128a800) d45j.sanityn proj=0 00000080:00000004:0.0:1787826353.344618:0:226253:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057d128a800: age 8161, max_age 8168 00800000:00000004:0.0:1787826353.344630:0:226253:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: age 8161, max_age 8168 00000100:00100000:0.0:1787826353.344673:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665472:192.168.203.156@tcp:41:rm.0 00000100:00100000:0.0:1787826353.345388:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.356012:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665472:192.168.203.156@tcp:41:rm.0 00800000:00000004:0.0:1787826353.356049:0:226253:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793351/1916558 objects 519512/534207 00000080:00000004:0.0:1787826353.356070:0:226253:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793351/1916558 objects 519512/534207 00000080:00000004:0.0:1787826353.356076:0:226253:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793351/1916558 objects 519512/534207 00000080:00200000:0.0:1787826353.356121:0:226253:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826353.356163:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826353.356298:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19a:0x0]) in obj [0x240000400:0x19a:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826353.356588:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.356597:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826353.356613:0:226253:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826353.356618:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.356627:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:0.0:1787826353.356645:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x198:0x0]) in obj [0x200000401:0x198:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826353.356654:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.356662:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826353.356674:0:226253:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826353.356677:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.356684:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826353.356732:0:226253:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200000:0.0:1787826353.356762:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148), inode mode 41ed mask 1 00000080:00200000:0.0:1787826353.356782:0:226253:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e58:0x0](ffff9057dfe11148), flags=0 00000080:00200000:0.0:1787826353.356787:0:226253:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e58:0x0](ffff9057dfe11148), intent=getattr 00000002:00010000:0.0:1787826353.356808:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x240000400:0x19a:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826353.356857:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.356865:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.356876:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.356900:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665600:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826353.356997:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.362480:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665600:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826353.362517:0:226253:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362550:0:226253:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362560:0:226253:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362574:0:226253:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362586:0:226253:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362603:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362612:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362620:0:226253:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.362633:0:226253:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332665600/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.362647:0:226253:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362659:0:226253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 8589935619/7616 (0 0) 00000002:00010000:0.0:1787826353.362677:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000403:0x1dc0:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826353.362754:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362763:0:226253:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.362774:0:226253:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226253 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.362798:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665728:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826353.362909:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.373608:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332665728:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826353.373642:0:226253:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 4/1,0 mode: --/CR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373658:0:226253:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373673:0:226253:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373686:0:226253:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373717:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373727:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373736:0:226253:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.373749:0:226253:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332665728/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.373764:0:226253:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373781:0:226253:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935619/7616 (0 0) 00000080:00200000:0.0:1787826353.373804:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x19a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826353.373850:0:226253:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dc0:0x0](ffff9057dfe12a08) 00000080:00010000:0.0:1787826353.373931:0:226253:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc0:0x0](ffff9057dfe12a08) for remote lock 0x95194e8760801a0d 00000080:00010000:0.0:1787826353.373937:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc0:0x0](ffff9057dfe12a08) for lock 0x95194e8760801a06 00000080:00010000:0.0:1787826353.373946:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dc0:0x0](ffff9057dfe12a08) 00000080:00010000:0.0:1787826353.373950:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a06 from it ffffb512c82bbdf8 00010000:00010000:0.0:1787826353.373955:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373966:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826353.373982:0:226253:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801a0d from it ffffb512c82bbdf8 00010000:00010000:0.0:1787826353.373986:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.373994:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826353.374017:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148), inode mode 41ed mask 3 00000080:00200000:0.0:1787826353.374026:0:226253:0:(namei.c:2418:ll_unlink()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e58:0x0](ffff9057dfe11148) 00010000:00010000:0.0:1787826353.374064:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.374112:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.374119:0:226253:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.374161:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801a06 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bf00 expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.374192:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.374306:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.374314:0:226253:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.374317:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801a0d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc0:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bf0e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826353.374340:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826353.374349:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c380 x1874663332666112/t0(0) o36->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826353.374372:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666112:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826353.374830:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:3.0:1787826353.379583:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826353.380192:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346310912 00000100:00100000:3.0:1787826353.380214:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346310912 00000100:00100000:3.0:1787826353.380310:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43660 00000100:00100000:3.0:1787826353.380321:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb220000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346310912:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.380343:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.380437:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826353.380444:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb220000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:8152:x1874663346310912:12345-192.168.203.156@tcp:104: Request processed in 125us (894us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.380453:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43660 00010000:00010000:3.0:1787826353.381096:0:6166:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826353.381105:0:6166:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381116:0:6166:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826353.381121:0:6166:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381136:0:6166:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381146:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381326:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381335:0:6166:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826353.381359:0:6166:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381368:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2d200/0x95194e87608019ea lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beac expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.381390:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826353.381393:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826353.381396:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.382679:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332665856:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826353.393596:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666112:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826353.393679:0:226253:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826353.393695:0:226253:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826353.393741:0:226253:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826353.393750:0:226253:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000403:0x1dc0:0x0](ffff9057dfe12a08) 00000080:00200008:0.0:1787826353.393969:0:226253:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e58:0x0](ffff9057dfe11148), flags 704000 00010000:00010000:0.0:1787826353.394005:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.394055:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826353.394076:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.394088:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.394100:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826353.394108:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.394115:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826353.394119:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:0.0:1787826353.394128:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c82bbc10 00010000:00010000:0.0:1787826353.394133:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826353.394142:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826353.394159:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826353.394170:0:226253:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=0 00000080:00200000:0.0:1787826353.394177:0:226253:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826353.394183:0:226253:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.394189:0:226253:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.394211:0:226253:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1787826353.394218:0:226253:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.394289:0:226253:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826353.394292:0:226253:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801981 00000080:00010000:0.0:1787826353.394296:0:226253:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801981 from it ffffb512c82bbd88 00010000:00010000:0.0:1787826353.394299:0:226253:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.394308:0:226253:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2c00/0x95194e8760801981 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bd4e expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1787826353.394316:0:226253:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826353.394323:0:226253:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988) 00010000:00010000:0.0:1787826353.394346:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.394402:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.394411:0:226253:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.394444:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9c00/0x95194e8760801988 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bd6a expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826353.394493:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332666368:192.168.203.156@tcp:103:rm.0 00010000:00010000:0.0:1787826353.394837:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:0.0:1787826353.394883:0:226253:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158f100 x1874663332665344/t4294985466(4294985466) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826353.394897:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826353.394903:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d180 x1874663332666624/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826353.394921:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666624:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826353.394951:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.405974:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332666368:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826353.407415:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666624:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826353.407465:0:226253:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158f100 x1874663332665344/t4294985466(4294985466) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787826353.407530:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.407552:0:226253:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.407596:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 1/0,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.407605:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.407611:0:226253:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.407617:0:226253:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787826353.407620:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e87608019ff lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e58:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bedd expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.407633:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8600/0x95194e87608019e3 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4be9e expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.407642:0:226253:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e876080198f lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4bd78 expref: -99 pid: 226215 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826353.407664:0:226253:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826353.407671:0:226253:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d180 x1874663332666880/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826353.407696:0:226253:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666880:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826353.407811:0:226253:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826353.420446:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826353.420490:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346312448 00000100:00100000:3.0:1787826353.420505:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346312448 00000100:00100000:3.0:1787826353.420592:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346312576 00000100:00100000:3.0:1787826353.420596:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346312576 00000100:00100000:3.0:1787826353.420622:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43661 00000100:00100000:3.0:1787826353.420635:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346312448:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.420657:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e87608019ab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bdb7 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826353.421176:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826353.421184:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346312448:12345-192.168.203.156@tcp:104: Request processed in 552us (8472us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.421192:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43661 00000100:00100000:0.0:1787826353.422984:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332667008:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.439513:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332667008:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826353.445888:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346313344 00000100:00100000:3.0:1787826353.445901:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346313344 00000100:00100000:3.0:1787826353.445975:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43663 00000100:00100000:3.0:1787826353.445986:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346313344:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.446008:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826353.446100:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826353.446108:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346313344:12345-192.168.203.156@tcp:104: Request processed in 123us (688us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.446115:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43663 00010000:00010000:0.0:1787826353.446550:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826353.446561:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.446571:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b07e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826353.446577:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.446587:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.446596:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.446684:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 2/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.446692:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.447126:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.447136:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b07e00/0x95194e87608019b2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e58:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bdd3 expref: -99 pid: 226217 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.447164:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.447166:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.447168:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826353.459125:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346313728 00000100:00100000:3.0:1787826353.459139:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346313728 00000100:00100000:3.0:1787826353.459217:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43664 00000100:00100000:3.0:1787826353.459261:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346313728:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.459287:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826353.459391:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826353.459399:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346313728:12345-192.168.203.156@tcp:104: Request processed in 141us (490us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.459410:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43664 00010000:00010000:0.0:1787826353.459615:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826353.459624:0:15672:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459634:0:15672:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3800 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826353.459639:0:15672:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459649:0:15672:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459659:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459799:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459807:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.459840:0:15672:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459848:0:15672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e87608019f1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x198:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4beba expref: -99 pid: 226253 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826353.459874:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.459876:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.459878:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.459897:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332667392:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.467022:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332667392:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.484628:0:226253:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226253:1874663332666880:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826353.484703:0:226253:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e58:0x0](ffff9057dfe11148) 00010000:00010000:0.0:1787826353.649478:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.649494:0:226266:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826353.649504:0:226266:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a14 from it ffffb512c82d3930 00010000:00010000:0.0:1787826353.649510:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.649907:0:226266:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.650025:0:226266:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826353.650046:0:226266:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.650056:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.650071:0:226266:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.650079:0:226266:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.650085:0:226266:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826353.650089:0:226266:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801a14 00000080:00010000:0.0:1787826353.650094:0:226266:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a14 from it ffffb512c82d3be8 00010000:00010000:0.0:1787826353.650097:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.650105:0:226266:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826353.650114:0:226266:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826353.650123:0:226266:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:0.0:1787826353.650151:0:226266:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826353.650156:0:226266:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826353.650163:0:226266:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.650170:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.650178:0:226266:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826353.650185:0:226266:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.650188:0:226266:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826353.650190:0:226266:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801a14 00000080:00010000:0.0:1787826353.650194:0:226266:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a14 from it ffffb512c82d3cb8 00010000:00010000:0.0:1787826353.650208:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826353.650215:0:226266:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826353.651312:0:226266:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826353.651325:0:226266:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826353.651347:0:226266:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.651422:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226266 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.651432:0:226266:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226266 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.651444:0:226266:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226266 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.651473:0:226266:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226266:1874663332667904:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826353.651871:0:226266:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826353.658076:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826353.659935:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332668032:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826353.662124:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.662133:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.662135:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.667537:0:226266:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057d128a800: committing for last_committed 8589957468 gen 1 00000100:00100000:0.0:1787826353.667591:0:226266:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226266:1874663332667904:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826353.667619:0:226266:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667636:0:226266:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667649:0:226266:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667662:0:226266:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667677:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667686:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667695:0:226266:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.667708:0:226266:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332667904/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.667925:0:226266:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.667942:0:226266:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826353.667949:0:226266:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826353.667986:0:226266:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826353.667994:0:226266:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801a22 00000080:00010000:0.0:1787826353.668000:0:226266:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826353.668004:0:226266:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a22 from it ffffb512c82d3cb8 00010000:00010000:0.0:1787826353.668008:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.668018:0:226266:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826353.668624:0:226266:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826353.668653:0:226266:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.668668:0:226266:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.668683:0:226266:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826353.668690:0:226266:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826353.668695:0:226266:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826353.668699:0:226266:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801a14 00000080:00010000:0.0:1787826353.668704:0:226266:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a14 from it ffffb512c82d3bb0 00010000:00010000:0.0:1787826353.668707:0:226266:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826353.668726:0:226266:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826353.668908:0:226266:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 44 00000080:00200008:0.0:1787826353.668920:0:226266:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200080:0.0:1787826353.668943:0:226266:0:(dir.c:2001:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) cmd=c03066f0 arg=7fffe5a92ff0 00000080:00200000:0.0:1787826353.668959:0:226266:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) name=d45j.sanityn stripe_offset=1 stripe_count=2, hash_type=3 00000020:00100000:0.0:1787826353.669116:0:226266:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826353.669129:0:226266:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332668288/t0(0) o36->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826353.669158:0:226266:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226266:1874663332668288:192.168.203.156@tcp:36:lfs.0 00000100:00100000:0.0:1787826353.669438:0:226266:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.674675:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332668032:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826353.687934:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43667 00000100:00100000:3.0:1787826353.687947:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346315904:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826353.687959:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826353.688596:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826353.688603:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346315904:12345-192.168.203.156@tcp:104: Request processed in 660us (1686us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826353.688610:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43667 00010000:00010000:0.0:1787826353.688757:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826353.688767:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688779:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3800 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826353.688787:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688798:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688809:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688892:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688899:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.688931:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688940:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801a22 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bfc4 expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.688965:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826353.688969:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.688976:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826353.688979:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.688986:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.688992:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.689061:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.689069:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.689075:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.689083:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2d400/0x95194e8760801a14 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4bf9a expref: -99 pid: 226266 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826353.689117:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.689119:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.689121:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.689145:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332668416:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.689310:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826353.689318:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332668544:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.724824:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332668416:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.724870:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332668544:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.804817:0:226254:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d8b0b180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.819053:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332668928:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826353.819086:0:226254:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819101:0:226254:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819116:0:226254:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819129:0:226254:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819143:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819153:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819161:0:226254:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.819173:0:226254:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb238380 x1874663332668928/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.819187:0:226254:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819211:0:226254:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7770 (0 0) 00000080:00200000:0.0:1787826353.819286:0:226254:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:0.0:1787826353.819373:0:226254:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.819436:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819445:0:226254:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.819466:0:226254:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.819499:0:226254:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669056:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826353.819609:0:226254:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.838111:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669056:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826353.838149:0:226254:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838164:0:226254:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838177:0:226254:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838190:0:226254:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838217:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838276:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838283:0:226254:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.838296:0:226254:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f100 x1874663332669056/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.838309:0:226254:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838324:0:226254:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/411 (0 0) 00010000:00010000:0.0:1787826353.838334:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838345:0:226254:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3800/0x95194e8760801a3e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c042 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826353.838380:0:226254:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826353.838445:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838453:0:226254:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.838463:0:226254:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.838485:0:226254:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669184:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826353.838592:0:226254:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.853652:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669184:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826353.853690:0:226254:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853725:0:226254:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853747:0:226254:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853760:0:226254:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853775:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853785:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853792:0:226254:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.853805:0:226254:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f100 x1874663332669184/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.853821:0:226254:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853837:0:226254:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/409 (0 0) 00010000:00010000:0.0:1787826353.853847:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853856:0:226254:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826353.853900:0:226254:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5a:0x0], updating i_size 8192 00000080:00200000:0.0:1787826353.853905:0:226254:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e5a:0x0](ffff9057dfe0db88): rc = 0 00000080:00010000:0.0:1787826353.853917:0:226254:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0db88) for remote lock 0x95194e8760801a37 00000080:00010000:0.0:1787826353.853922:0:226254:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0db88) for lock 0x95194e8760801a30 00000080:00010000:0.0:1787826353.853938:0:226254:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0db88) 00000080:00010000:0.0:1787826353.853943:0:226254:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a30 from it ffffb512c829bac8 00010000:00010000:0.0:1787826353.853946:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c800/0x95194e8760801a30 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c026 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853954:0:226254:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c800/0x95194e8760801a30 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c026 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826353.853961:0:226254:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801a37 from it ffffb512c829bac8 00010000:00010000:0.0:1787826353.853965:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.853973:0:226254:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057edd2de00/0x95194e8760801a37 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c034 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826353.854000:0:226254:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0db88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826353.854044:0:226254:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0db88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826353.854051:0:226254:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5a:0x0](ffff9057dfe0db88), file ffff9057c6354780, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826353.854061:0:226254:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5a:0x0](ffff9057dfe0db88), intent=open|creat 00000002:00010000:0.0:1787826353.854092:0:226254:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x200000403:0x1dc1:0x0]) in obj [0x200000401:0x199:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826353.854115:0:226254:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.854175:0:226254:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.854183:0:226254:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826353.854186:0:226254:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801a45 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c050 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826353.854329:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 3/0,1 mode: --/CW res: [0x200000401:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.854337:0:226254:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 3/0,1 mode: --/CW res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826353.854350:0:226254:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826353.854358:0:226254:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f100 x1874663332669440/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826353.854368:0:226254:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 3/0,1 mode: --/CW res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226254 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826353.854392:0:226254:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669440:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826353.854586:0:226254:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.873170:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332669440:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826353.873219:0:226254:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/0,1 mode: --/CW res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873283:0:226254:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873293:0:226254:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873306:0:226254:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873324:0:226254:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873339:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873350:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873357:0:226254:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826353.873371:0:226254:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158f100 x1874663332669440/t8589957471(8589957471) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826353.873385:0:226254:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f100 x1874663332669440/t8589957471(8589957471) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826369 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826353.873400:0:226254:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826353.873414:0:226254:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.873426:0:226254:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7617 (0 0) 00000080:00200000:0.0:1787826353.873433:0:226254:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000401:0x199:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826353.873449:0:226254:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5a:0x0](ffff9057dfe0db88), intent=open|creat 00000080:00200000:0.0:1787826353.873634:0:226254:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dc1:0x0](ffff9057dfe11988) 00000100:00100000:0.0:1787826353.929942:0:226254:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973859 gen 19 00000100:00100000:0.0:1787826353.930057:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332670208:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826353.930086:0:226254:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.930109:0:226254:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.930123:0:226254:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.930141:0:226254:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826353.930158:0:226254:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.930170:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.930178:0:226254:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826353.930689:0:226254:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn-2: write [0, 6) append 0 append lockpos 0 00000080:00200000:0.0:1787826353.930880:0:226254:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826353.930949:0:226254:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:0.0:1787826353.930979:0:226254:0:(rw26.c:929:ll_write_end()) page@00000000957dd3ed[3 000000002042607b 2 1 00000000b5715f5d 0 0 0] 00000080:00200000:0.0:1787826353.930983:0:226254:0:(rw26.c:929:ll_write_end()) vmpage @00000000a432f73c 17ffffc0001001 3:0 ffff9057fb2ba7e0 0 lru 00000080:00200000:0.0:1787826353.930991:0:226254:0:(rw26.c:929:ll_write_end()) osc-page@0000000058418186 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 0000000072dd0513 000000004654729d > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787826353.930993:0:226254:0:(rw26.c:929:ll_write_end()) end page@00000000957dd3ed 00000080:00200000:0.0:1787826353.930995:0:226254:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787826353.931009:0:226254:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:0.0:1787826353.931038:0:226254:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826353.931061:0:226254:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfe08ab8, count 1, dirtied 1 00000080:00200000:0.0:1787826353.931067:0:226254:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:0.0:1787826353.931070:0:226254:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn-2: write bytes 0, result: 6 00000020:00010000:0.0:1787826353.931089:0:226254:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:0.0:1787826353.931094:0:226254:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826353.931097:0:226254:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9bb2730 (ffff9057d889cc68/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826353.931103:0:226254:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826353.931114:0:226254:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826353.931133:0:226254:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2e48 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826353.931136:0:226254:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9bb2730 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826353.931147:0:226254:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826353.931155:0:226254:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787826353.931165:0:226254:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e5b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826353.931169:0:226254:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn-2: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826353.931177:0:226254:0:(file.c:1775:ll_merge_attr_nolock()) [0x240000403:0x1e5b:0x0] updating i_size 6 i_blocks 0 00000080:00200000:0.0:1787826353.931182:0:226254:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:0.0:1787826353.931260:0:226254:0:(file.c:479:ll_file_release()) START file f45j.sanityn-2:[0x240000403:0x1e5b:0x0](ffff9057dfe08908), flags 100001 00000002:00100000:0.0:1787826353.931360:0:226254:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb239180 x1874663332670080/t4294985471(4294985471) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826369 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826353.931373:0:226254:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826353.931559:0:226254:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c380 x1874663332670464/t0(0) o35->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826353.931589:0:226254:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332670464:192.168.203.156@tcp:35:bash.0 00000100:00100000:0.0:1787826353.931704:0:226254:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826353.937800:0:226254:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226254:1874663332670464:192.168.203.156@tcp:35:bash.0 00000002:00100000:0.0:1787826353.937851:0:226254:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb239180 x1874663332670080/t4294985471(4294985471) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826369 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787826353.960769:0:226269:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826353.960814:0:226269:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826353.960838:0:226269:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/6 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1787826353.962245:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826353.962253:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826353.962269:0:15673:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e59:0x0]: losing layout lock 00010000:00010000:0.0:1787826353.962297:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 1/0,0 mode: --/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.962305:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826353.962314:0:15673:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:398e0287-d3c7-44dd-b1d9-6b85f698e615:15673:1874663332670848:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826353.962360:0:15673:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00000100:00100000:3.0:1787826353.968105:0:6166:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3b80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:6166:1874663332670592:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826353.968120:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2ce00/0x95194e8760801a29 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c00a expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826353.968135:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801a4c lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c06c expref: -99 pid: 226254 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826353.968148:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2c800/0x95194e8760801a30 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c026 expref: -99 pid: 226254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826353.968177:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826353.968179:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826353.968181:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826353.968950:0:15673:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:398e0287-d3c7-44dd-b1d9-6b85f698e615:15673:1874663332670848:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826353.968969:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e87608019dc lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e59:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4be89 expref: -99 pid: 226215 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826353.969006:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.969008:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826353.969011:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826354.607080:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332670976:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607213:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607263:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607283:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332671744:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607301:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332672000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607327:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:3.0:1787826354.607332:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671104:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607348:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607364:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332671616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.607386:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332671872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.647595:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.647622:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332671616:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.647635:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332671488:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.647647:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332672000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826354.656053:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332671872:192.168.203.156@tcp:400:kworker.0 00000080:00200000:3.0:1787826355.620487:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826355.620510:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826355.620543:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826355.620558:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826355.620563:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826355.620570:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826355.620576:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826355.620579:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826355.620600:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009b027951 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000935242d3/0x95194e8760801a53 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c07a expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826355.620622:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009b027951 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000935242d3/0x95194e8760801a53 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c07a expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826355.620692:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000935242d3/0x95194e8760801a53 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c07a expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826355.620801:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb222d80 - object 0x2c0000401:1315 offset 0<>6 00000100:00100000:3.0:1787826355.620895:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332672256:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826355.621009:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826355.621015:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332672128:192.168.203.156@tcp:4:bash.0 00000080:00200000:0.0:1787826355.649941:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826355.649962:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826355.649975:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787826355.649998:0:149405:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826355.650012:0:149405:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826355.650264:0:149405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1787826355.650336:0:149405:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158e680 - object 0x280000400:1232 offset 0<>6 00000100:00100000:0.0:1787826355.650412:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332672384:192.168.203.156@tcp:4:bash.0 00000100:00100000:0.0:1787826355.650496:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826355.650503:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332672512:192.168.203.156@tcp:16:kworker.0 00000100:00100000:0.0:1787826355.659424:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332672512:192.168.203.156@tcp:16:kworker.0 00000008:00010000:0.0:1787826355.659478:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787826355.659521:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000009fad5af9 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787826355.659555:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332672384:192.168.203.156@tcp:4:bash.0 00000080:00200000:3.0:1787826355.659607:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1787826356.002592:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826356.131382:0:149405:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dbc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826356.131404:0:149405:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826356.131441:0:149405:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dbc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1787826357.026406:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826357.870297:0:226435:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332673408:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826357.870571:0:226435:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51180 going to sleep for 11 seconds 00010000:00010000:0.0:1787826358.050371:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826358.439271:0:226435:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332673408:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826358.439299:0:226435:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439319:0:226435:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439327:0:226435:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439338:0:226435:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439349:0:226435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439364:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439379:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439387:0:226435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826358.439400:0:226435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb23b100 x1874663332673408/t8589957473(8589957473) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826373 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826358.439414:0:226435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb23b100 x1874663332673408/t8589957473(8589957473) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826373 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826358.439427:0:226435:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826358.439448:0:226435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.439460:0:226435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935619/7617 (0 0) 00000080:00200000:0.0:1787826358.439466:0:226435:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000401:0x199:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826358.439497:0:226435:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dc1:0x0](ffff9057dfe13248) 00000080:00200000:0.0:1787826358.439570:0:226435:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dc1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826358.439576:0:226435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc1:0x0](ffff9057dfe13248) for lock 0x95194e8760801a8b 00000080:00010000:0.0:1787826358.439587:0:226435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dc1:0x0](ffff9057dfe13248) 00000080:00200008:0.0:1787826358.439601:0:226435:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1dc1:0x0](ffff9057dfe13248), flags 100000 00000002:00100000:0.0:1787826358.439624:0:226435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb23b100 x1874663332673408/t8589957473(8589957473) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826373 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826358.439989:0:226435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a8b from it ffff9057f9ac5180 00010000:00010000:0.0:1787826358.439996:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.440005:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826358.440031:0:226435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1dc1:0x0](ffff9057dfe13248), inode mode 81a4 mask 44 00000080:00200008:0.0:1787826358.440054:0:226435:0:(file.c:6296:ll_getattr_dentry()) START file f45j.sanityn:[0x200000403:0x1dc1:0x0](ffff9057dfe13248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826358.440061:0:226435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000403:0x1dc1:0x0](ffff9057dfe13248),name=f45j.sanityn 00000002:00010000:0.0:1787826358.440080:0:226435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dc1:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826358.440091:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 2/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.440101:0:226435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 2/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826358.440108:0:226435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x199:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826358.440114:0:226435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dc1:0x0](ffff9057dfe13248) 00000080:00010000:0.0:1787826358.440117:0:226435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc1:0x0](ffff9057dfe13248) for lock 0x95194e8760801a8b 00000080:00010000:0.0:1787826358.440121:0:226435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a8b from it ffffb512c8473cd8 00010000:00010000:0.0:1787826358.440124:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.440132:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826358.440153:0:226435:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826358.440171:0:226435:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc1:0x0] 00000008:00010000:0.0:1787826358.440188:0:226435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826358.440191:0:226435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d930, osc lock ffff9057c8b82000, flags 0x400001002 00010000:00010000:0.0:1787826358.440259:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.440270:0:226435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826358.440303:0:226435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826358.440318:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826358.440335:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332674560:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826358.445201:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826358.446716:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826358.446725:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446736:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c800 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826358.446741:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446751:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446760:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446826:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446833:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826358.446854:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446862:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801a7d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0ea expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.446882:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826358.446884:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826358.446886:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826358.446904:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332674688:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826358.450758:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826358.457456:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332674688:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826358.461514:0:2393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461535:0:2393:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461550:0:2393:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461558:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461566:0:2393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826358.461580:0:2393:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 6, setting rss=6; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1787826358.461587:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826358.461605:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461614:0:2393:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461623:0:2393:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 4/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461630:0:2393:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2e00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826358.461636:0:2393:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 5/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461647:0:2393:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826358.461653:0:2393:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 3/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.461661:0:2393:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000b86cd977/0x95194e8760801ab5 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1787826358.461820:0:226435:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc1:0x0] updating i_size 6 i_blocks 8 00000020:00010000:3.0:1787826358.461827:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_release():245 00000020:00010000:3.0:1787826358.461830:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_cancel():142 00000020:00010000:3.0:1787826358.461836:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d930 (ffff9057d889cc68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1787826358.461840:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89030 (ffff9057d889c908/0) at cl_lock_fini():72 00000020:00010000:3.0:1787826358.461842:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d930 (ffff9057d889c908/1) at cl_lock_fini():72 00000080:00200000:3.0:1787826358.461856:0:226435:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787826358.462075:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332674560:192.168.203.156@tcp:101:cat.0 00000080:00200008:3.0:1787826358.463291:0:226435:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dc1:0x0], ppos: 0, count: 4194304 00000080:00200000:3.0:1787826358.463317:0:226435:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:3.0:1787826358.463333:0:226435:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 0, bytes: 4194304 00000080:00200000:3.0:1787826358.463339:0:226435:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1787826358.463347:0:226435:0:(cl_io.c:214:cl_io_rw_init()) header@00000000866acd84[0x110, 1, [0x200000403:0x1dc1:0x0] hash] 00000020:00200000:3.0:1787826358.463349:0:226435:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:3.0:1787826358.463353:0:226435:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826358.463373:0:226435:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:3.0:1787826358.463377:0:226435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826358.463395:0:226435:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:3.0:1787826358.463411:0:226435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1787826358.463417:0:226435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057f9a713b0, osc lock ffff9057c50bb560, flags 0x0 00010000:00010000:3.0:1787826358.463470:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.463482:0:226435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.463492:0:226435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826358.463515:0:226435:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332674816:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826358.463603:0:226435:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d840 going to sleep for 11 seconds 00000100:00100000:3.0:1787826358.470562:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826358.476835:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826358.478974:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826358.478993:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332675072:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826358.484291:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332674944:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826358.499836:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826358.499884:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826358.499962:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346323200 00000100:00100000:3.0:1787826358.499971:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346323200 00000100:00100000:3.0:1787826358.500013:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346323328 00000100:00100000:3.0:1787826358.500017:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346323328 00000100:00100000:3.0:1787826358.500033:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43672 00000100:00100000:3.0:1787826358.500043:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb223480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346323200:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826358.500061:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801a8b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4c18b expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826358.500328:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826358.500336:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb223480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6542:x1874663346323200:12345-192.168.203.156@tcp:104: Request processed in 294us (509us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826358.500344:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43672 00000100:00100000:3.0:1787826358.500363:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43673 00000100:00100000:3.0:1787826358.500370:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb221180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6511:x1874663346323328:12345-192.168.203.156@tcp:105: 00010000:00010000:3.0:1787826358.500381:0:6159:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500391:0:6159:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500402:0:6159:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500421:0:6159:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000000000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500442:0:6159:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000000000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500453:0:6159:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x80000000000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826358.500472:0:6159:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9058021f9e00) 00000100:00100000:3.0:1787826358.500496:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb221180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6511:x1874663346323328:12345-192.168.203.156@tcp:105: Request processed in 127us (611us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826358.500503:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43673 00000100:00100000:0.0:1787826358.501027:0:226435:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332674816:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826358.501046:0:226435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.501063:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826358.501075:0:226435:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.501087:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.501096:0:226435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826358.501134:0:226435:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826358.501157:0:226435:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc1:0x0] 00000008:00010000:0.0:1787826358.501179:0:226435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826358.501183:0:226435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d930, osc lock ffff9057c8b83c80, flags 0x400001002 00010000:00010000:0.0:1787826358.501199:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.501212:0:226435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826358.501256:0:226435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826358.501259:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.501269:0:226435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826358.501276:0:226435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826358.501283:0:226435:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc1:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826358.501287:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826358.501290:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.501293:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d930 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.501297:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826358.501300:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d930 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826358.501309:0:226435:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1787826358.501351:0:226435:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff9057c644b480 00000008:00010000:0.0:1787826358.501383:0:215335:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9057c644b480 00010000:00010000:0.0:1787826358.501404:0:215335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00100000:0.0:1787826358.501454:0:215335:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158ce00 - object 0x2c0000401:1315 offset 0<>4096 00000100:00100000:0.0:1787826358.501494:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332675200:192.168.203.156@tcp:3:cat.0 00000020:00010000:0.0:1787826358.514883:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826358.514888:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.514892:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057f9a713b0 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826358.514902:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.514917:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826358.514932:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826358.514935:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057f9a713b0 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826358.514943:0:226435:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826358.514955:0:226435:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826358.514959:0:226435:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826358.514964:0:226435:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:0.0:1787826358.514983:0:226435:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dc1:0x0], ppos: 6, count: 4194304 00000080:00200000:0.0:1787826358.514991:0:226435:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:0.0:1787826358.514995:0:226435:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826358.515000:0:226435:0:(cl_io.c:214:cl_io_rw_init()) header@00000000866acd84[0x110, 2, [0x200000403:0x1dc1:0x0] hash] 00000020:00200000:0.0:1787826358.515002:0:226435:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:0.0:1787826358.515005:0:226435:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826358.515031:0:226435:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000403:0x1dc1:0x0](ffff9057dfe13248) 00000002:00010000:0.0:1787826358.515040:0:226435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dc1:0x0]) in obj [0x200000403:0x1dc1:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826358.515083:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.515092:0:226435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.515102:0:226435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226435 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826358.515119:0:226435:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332675456:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826358.515205:0:226435:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826358.519554:0:226435:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332675456:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826358.519583:0:226435:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.519596:0:226435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826358.519610:0:226435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158c700 x1874663332675456/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826374 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826358.519620:0:226435:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826358.519632:0:226435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.519644:0:226435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 16 res 8589935619/7617 (0 0) 00000080:00200000:0.0:1787826358.519650:0:226435:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000403:0x1dc1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826358.519665:0:226435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc1:0x0](ffff9057dfe13248) for lock 0x95194e8760801ac3 00000080:00010000:0.0:1787826358.519673:0:226435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801ac3 from it ffffb512c8473af8 00010000:00010000:0.0:1787826358.519677:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826358.519692:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519714:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519724:0:226435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826358.519731:0:226435:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000403:0x1dc1:0x0](00000000dc35cfa5) being reconfigured ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826358.519767:0:226435:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dc1:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826358.519772:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519779:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826358.519801:0:226435:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:0.0:1787826358.519805:0:226435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826358.519823:0:226435:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:0.0:1787826358.519837:0:226435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826358.519841:0:226435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5dbb0, osc lock ffff9057c8b83c80, flags 0x0 00010000:00010000:0.0:1787826358.519856:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519867:0:226435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519878:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519886:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519893:0:226435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826358.519901:0:226435:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:0.0:1787826358.519916:0:226435:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc1:0x0] 00000008:00010000:0.0:1787826358.519921:0:226435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826358.519923:0:226435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d1b0, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826358.519929:0:226435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519937:0:226435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826358.519945:0:226435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826358.519948:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.519974:0:226435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826358.519981:0:226435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826358.519987:0:226435:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc1:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826358.519990:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826358.519993:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.519996:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d1b0 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.519999:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826358.520002:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d1b0 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826358.520009:0:226435:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:0.0:1787826358.520022:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826358.520024:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826358.520027:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5dbb0 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826358.520031:0:226435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826358.520040:0:226435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826358.520050:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826358.520052:0:226435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5dbb0 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826358.520058:0:226435:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826358.520068:0:226435:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826358.520072:0:226435:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:0.0:1787826358.520075:0:226435:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:0.0:1787826358.520187:0:226435:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dc1:0x0](ffff9057dfe13248), flags 100000 00000002:00100000:0.0:1787826358.520288:0:226435:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb23b100 x1874663332673408/t8589957473(8589957473) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826373 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826358.520299:0:226435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826358.520304:0:226435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332675712/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826358.520323:0:226435:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332675712:192.168.203.156@tcp:35:cat.0 00000100:00100000:0.0:1787826358.520402:0:226435:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826358.530885:0:226435:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226435:1874663332675712:192.168.203.156@tcp:35:cat.0 00000002:00100000:0.0:1787826358.530924:0:226435:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb23b100 x1874663332673408/t8589957473(8589957473) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826373 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826358.970672:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826358.972115:0:6166:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826358.972125:0:6166:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826358.972136:0:6166:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9e00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826358.972142:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800428400030000 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826358.972175:0:6166:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826358.972272:0:6166:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826358.972318:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826358.972330:0:6166:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826358.972351:0:6166:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826358.972358:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000061d2a4ef/0x95194e8760801abc lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c1bc expref: -99 pid: 226435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826358.972379:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826358.972381:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826358.972383:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826358.972390:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826358.972405:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332675840:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826358.981211:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332675840:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826359.075016:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826359.714419:0:160397:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1787826359.714436:0:160397:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9057d128a800: disconnect after 21s idle 00000100:00100000:3.0:1787826359.746960:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826359.746983:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332676480:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826359.747201:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332676608:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826359.747274:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826359.747281:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332676736:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826359.747329:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332676864:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826359.747378:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826359.747384:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332676992:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826359.759543:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332676480:192.168.203.156@tcp:400:kworker.0 00000100:00080000:0.0:1787826359.761909:0:2392:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9057fb222a00 x1874663332676352/t0(0) o9->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 0 dl 1787826369 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826359.761921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1787826359.761926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332676352:192.168.203.156@tcp:9:kworker.0 00000080:00200000:0.0:1787826359.784034:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826359.784062:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.784080:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.784097:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826359.784107:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826359.784116:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826359.784119:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801a68 00000080:00010000:0.0:1787826359.784128:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a68 from it ffffb512c846bb58 00010000:00010000:0.0:1787826359.784132:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.784141:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826359.784172:0:226473:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826359.784180:0:226473:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:0.0:1787826359.784189:0:226473:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826359.784194:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248),name=d45j.sanityn 00000002:00010000:0.0:1787826359.784207:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.784214:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7770 (0 0) 00010000:00010000:0.0:1787826359.784291:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.784299:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.784307:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826359.784331:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677120:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826359.784405:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51b40 going to sleep for 11 seconds 00000100:00100000:3.0:1787826359.794912:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677120:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826359.794938:0:226473:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.794960:0:226473:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.794968:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.794979:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.794990:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795003:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795012:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795019:0:226473:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826359.795033:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332677120/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826359.795047:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795058:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7770 (0 0) 00000080:00200000:3.0:1787826359.795064:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826359.795096:0:226473:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5a:0x0], updating i_size 4096 00000080:00010000:3.0:1787826359.795102:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) for lock 0x95194e8760801aca 00000080:00010000:3.0:1787826359.795107:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) 00000080:00010000:3.0:1787826359.795111:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801aca from it ffffb512c846bcb8 00010000:00010000:3.0:1787826359.795115:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795123:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1787826359.795161:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826359.795210:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795217:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.795248:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826359.795271:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677248:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826359.795340:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d780 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.803045:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677248:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826359.803069:0:226473:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803082:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803093:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803104:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803116:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803124:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803133:0:226473:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826359.803144:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223b80 x1874663332677248/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826359.803172:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803184:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/411 (0 0) 00010000:00010000:0.0:1787826359.803192:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803199:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826359.803242:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.803296:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803303:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.803312:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826359.803331:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677376:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826359.803408:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.806840:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677376:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826359.806868:0:226473:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806890:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806901:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806911:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806923:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806931:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806937:0:226473:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826359.806948:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ea00 x1874663332677376/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826359.806962:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806975:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/409 (0 0) 00010000:00010000:0.0:1787826359.806984:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.806992:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826359.807081:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826359.807100:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.807111:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.807126:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826359.807133:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826359.807139:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826359.807142:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801a68 00000080:00010000:0.0:1787826359.807148:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a68 from it ffffb512c846bbf8 00010000:00010000:0.0:1787826359.807151:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.807169:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826359.807178:0:226473:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826359.807184:0:226473:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826359.807191:0:226473:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), inode mode 41ed mask 44 00000080:00200008:0.0:1787826359.807198:0:226473:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), flags 704400 00000002:00010000:0.0:1787826359.807213:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5a:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1787826359.807280:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.807288:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826359.807298:0:226473:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826359.807305:0:226473:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ea00 x1874663332677632/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826359.807320:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826359.807338:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677632:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826359.807510:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.814836:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677632:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826359.814858:0:226473:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814884:0:226473:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814892:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814900:0:226473:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814909:0:226473:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814915:0:226473:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2a00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826359.814920:0:226473:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814927:0:226473:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814935:0:226473:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814944:0:226473:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826359.814950:0:226473:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.814957:0:226473:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801adf lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826359.814981:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057f158ea00 x1874663332677632/t4294985475(4294985475) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826359.814994:0:226473:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ea00 x1874663332677632/t4294985475(4294985475) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826359.815010:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826359.815041:0:226473:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5a:0x0], updating i_size 4096 00000002:00100000:0.0:1787826359.815060:0:226473:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158ea00 x1874663332677632/t4294985475(4294985475) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826359.815127:0:226473:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826359.815133:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248),name=d45j.sanityn 00000002:00010000:0.0:1787826359.815165:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.815179:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815190:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826359.815199:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.815205:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815213:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815239:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815250:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826359.815264:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.815273:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815281:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815289:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815296:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826359.815305:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826359.815310:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) 00000080:00010000:0.0:1787826359.815313:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) for lock 0x95194e8760801aca 00000080:00010000:0.0:1787826359.815318:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801aca from it ffffb512c846bcd8 00010000:00010000:0.0:1787826359.815321:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.815327:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826359.815339:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.815344:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815351:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815358:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815365:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826359.815372:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.815376:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815383:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815389:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.815396:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826359.815468:0:226473:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826359.815482:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826359.815503:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826359.815510:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826359.815517:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826359.816124:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677760:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826359.816250:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.832847:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677760:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826359.833635:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826359.833654:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826359.833683:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826359.833701:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826359.833716:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826359.834327:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677888:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826359.834440:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.844468:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332677888:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826359.845756:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826359.845776:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826359.845972:0:226473:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), flags 704000 00000002:00100000:0.0:1787826359.846056:0:226473:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158ea00 x1874663332677632/t4294985475(4294985475) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826375 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826359.846071:0:226473:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826359.846076:0:226473:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d500 x1874663332678144/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826359.846103:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678144:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826359.846246:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.854561:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678144:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826359.854612:0:226473:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158ea00 x1874663332677632/t4294985475(4294985475) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826359.854722:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826359.854745:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.854767:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.854784:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826359.854793:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826359.854799:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826359.854803:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801a68 00000080:00010000:0.0:1787826359.854812:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801a68 from it ffffb512c846bbf8 00010000:00010000:0.0:1787826359.854816:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.854825:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826359.854841:0:226473:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826359.854849:0:226473:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826359.854858:0:226473:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), inode mode 41ed mask 44 00000080:00200008:0.0:1787826359.854868:0:226473:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), flags 704400 00000002:00010000:0.0:1787826359.854885:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5a:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826359.854946:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.854954:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826359.854967:0:226473:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826359.854975:0:226473:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d500 x1874663332678400/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826359.854991:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826359.855013:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678400:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826359.855333:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.861474:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678400:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826359.861530:0:226473:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.861544:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.861558:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.861571:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826359.861588:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158d500 x1874663332678400/t4294985477(4294985477) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826359.861600:0:226473:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158d500 x1874663332678400/t4294985477(4294985477) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826359.861615:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.861627:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7770 (0 0) 00000080:00200000:0.0:1787826359.861635:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826359.861674:0:226473:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5a:0x0], updating i_size 4096 00000080:00010000:0.0:1787826359.861682:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) for lock 0x95194e8760801ae6 00000080:00010000:0.0:1787826359.861690:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801ae6 from it ffffb512c846bbd0 00010000:00010000:0.0:1787826359.861694:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826359.861705:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801ae6 lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c23a expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826359.861734:0:226473:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158d500 x1874663332678400/t4294985477(4294985477) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826375 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826359.861826:0:226473:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5a:0x0](ffff9057dfe0b248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826359.861834:0:226473:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248),name=d45j.sanityn 00000002:00010000:0.0:1787826359.861858:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.861870:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.861883:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826359.861892:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.861899:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861906:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861921:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861929:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826359.861939:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.861947:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861956:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861964:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826359.861972:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826359.861982:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826359.861987:0:226473:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) 00000080:00010000:0.0:1787826359.861991:0:226473:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5a:0x0](ffff9057dfe0b248) for lock 0x95194e8760801aca 00000080:00010000:0.0:1787826359.861994:0:226473:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801aca from it ffffb512c846bcd8 00010000:00010000:0.0:1787826359.861997:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826359.862004:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3000/0x95194e8760801aca lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c202 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826359.862017:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19b:0x0]) in obj [0x240000400:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.862023:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862032:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862040:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862047:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826359.862055:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x199:0x0]) in obj [0x200000401:0x199:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826359.862364:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862374:0:226473:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862383:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826359.862391:0:226473:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826359.862486:0:226473:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057d128a800) d45j.sanityn proj=0 00000080:00000004:0.0:1787826359.862492:0:226473:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057d128a800: age 8169, max_age 8174 00800000:00000004:0.0:1787826359.862504:0:226473:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: age 8169, max_age 8174 00000100:00100000:0.0:1787826359.862555:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678528:192.168.203.156@tcp:41:rm.0 00000100:00100000:0.0:1787826359.862659:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b513c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826359.871085:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678528:192.168.203.156@tcp:41:rm.0 00800000:00000004:0.0:1787826359.871121:0:226473:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793351/1916558 objects 519514/534209 00000080:00000004:0.0:1787826359.871129:0:226473:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793351/1916558 objects 519514/534209 00000080:00000004:0.0:1787826359.871134:0:226473:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793351/1916558 objects 519514/534209 00000080:00200000:0.0:1787826359.871189:0:226473:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e5a:0x0](ffff9057dfe0b248) pos/size0/8192 32bit_api 0 00000100:00100000:3.0:1787826359.880121:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678656:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826359.880167:0:226473:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: --/CR res: [0x200000401:0x199:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880194:0:226473:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880203:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880217:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880276:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880290:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880299:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880307:0:226473:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826359.880325:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de1180 x1874663332678656/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826359.880338:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801aed lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c25d expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880348:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7771 (0 0) 00000002:00010000:3.0:1787826359.880368:0:226473:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e5b:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826359.880424:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880431:0:226473:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.880441:0:226473:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226473 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826359.880464:0:226473:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678784:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826359.880655:0:226473:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826359.893784:0:226473:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226473:1874663332678784:192.168.203.156@tcp:101:rm.0 00010000:00010000:3.0:1787826359.893937:0:226473:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.893954:0:226473:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.893967:0:226473:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.893979:0:226473:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.893993:0:226473:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.894002:0:226473:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.894009:0:226473:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826359.894022:0:226473:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220380 x1874663332678784/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826375 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826359.894036:0:226473:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801af4 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c26b expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826359.894051:0:226473:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7771 (0 0) 00000080:00200000:3.0:1787826359.894076:0:226473:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x199:0x0] suppgids 0 -1: rc 0 00000100:00100000:3.0:1787826359.900012:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332678912:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826359.904128:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346326272 00000100:00100000:3.0:1787826359.904141:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346326272 00000100:00100000:3.0:1787826359.904303:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43675 00000100:00100000:3.0:1787826359.904316:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346326272:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826359.904340:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ad8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x199:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c21e expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826359.904446:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826359.904453:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346326272:12345-192.168.203.156@tcp:104: Request processed in 140us (564us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826359.904461:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43675 00000100:00100000:0.0:1787826359.919909:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332679424:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826359.922791:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332679424:192.168.203.156@tcp:103:rm.0 00010000:00010000:0.0:1787826359.930065:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826359.930077:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.930090:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c000 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826359.930096:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.930106:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.930117:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.930200:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.930209:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826359.932926:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.932936:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c000/0x95194e8760801a68 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c0b2 expref: -99 pid: 226435 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826359.932955:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.932957:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.932958:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826359.943876:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346328704 00000100:00100000:3.0:1787826359.943889:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346328704 00000100:00100000:3.0:1787826359.943980:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43677 00000100:00100000:3.0:1787826359.943992:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346328704:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826359.944013:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826359.944141:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826359.944148:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:8152:x1874663346328704:12345-192.168.203.156@tcp:104: Request processed in 159us (389us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826359.944170:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43677 00010000:00010000:0.0:1787826359.944853:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826359.944864:0:34880:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.944875:0:34880:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826359.944881:0:34880:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.944891:0:34880:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.944901:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.944981:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 2/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.944995:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826359.945342:0:34880:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.945352:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06600/0x95194e8760801a99 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c137 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826359.945371:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.945373:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.945375:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826359.949468:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346328960 00000100:00100000:3.0:1787826359.949479:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346328960 00000100:00100000:3.0:1787826359.949641:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43678 00000100:00100000:3.0:1787826359.949652:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35094:x1874663346328960:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826359.949679:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826359.950023:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826359.950032:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:35094:x1874663346328960:12345-192.168.203.156@tcp:104: Request processed in 381us (896us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826359.950043:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43678 00010000:00010000:0.0:1787826359.950403:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826359.950412:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950610:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f8e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826359.950617:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950627:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950637:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950824:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950839:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826359.950877:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950885:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801ad1 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19b:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c210 expref: -99 pid: 226473 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826359.950904:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.950906:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826359.950908:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826359.950918:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826359.950935:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332680448:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826359.953115:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332680448:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826360.082642:0:226481:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3480 pname:cluuid:pid:xid:nid:opc:job stat:fc0b407b-a541-420d-8712-b9c8da18fe8c:226481:1874663332680576:192.168.203.156@tcp:41:stat.0 00800000:00000004:0.0:1787826360.082681:0:226481:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057c7539000: update ffff9057c6ef0450 cache blocks 1793351/1916558 objects 519514/534206 00000080:00000004:0.0:1787826360.082691:0:226481:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057c7539000: update ffff9057c6ef37c0 cache blocks 1793351/1916558 objects 519514/534206 00000080:00000004:0.0:1787826360.082697:0:226481:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793351/1916558 objects 519514/534206 00010000:00010000:3.0:1787826360.100933:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826360.157632:0:226485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826360.157662:0:226485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826360.157725:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.157736:0:226485:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.157747:0:226485:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.157773:0:226485:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226485:1874663332680704:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826360.159566:0:226485:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.167588:0:226485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226485:1874663332680704:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826360.167615:0:226485:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167628:0:226485:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167638:0:226485:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167649:0:226485:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167662:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167677:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167686:0:226485:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.167698:0:226485:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158df80 x1874663332680704/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826360.167721:0:226485:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167734:0:226485:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826360.167740:0:226485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826360.167768:0:226485:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826360.167774:0:226485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801afb 00000080:00010000:0.0:1787826360.167779:0:226485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826360.167783:0:226485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801afb from it ffffb512c848bb58 00010000:00010000:0.0:1787826360.167954:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.167963:0:226485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.167988:0:226485:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826360.168008:0:226485:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:0.0:1787826360.168012:0:226485:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:0.0:1787826360.168032:0:226485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826360.168070:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.168077:0:226485:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.168086:0:226485:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.168107:0:226485:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226485:1874663332680832:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826360.168187:0:226485:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.200295:0:226485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226485:1874663332680832:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826360.200316:0:226485:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200329:0:226485:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200335:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200342:0:226485:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200362:0:226485:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200369:0:226485:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826360.200373:0:226485:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200379:0:226485:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200385:0:226485:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200391:0:226485:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826360.200399:0:226485:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.200405:0:226485:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b02 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226485 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.200423:0:226485:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057f158df80 x1874663332680832/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826360.200437:0:226485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826360.200464:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.200476:0:226485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826360.200484:0:226485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801afb from it ffffb512c848b930 00010000:00010000:0.0:1787826360.200487:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.200495:0:226485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826360.200565:0:226485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826360.200578:0:226485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826360.200586:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826360.200594:0:226485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826360.200607:0:226485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826360.200611:0:226485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826360.200615:0:226485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801afb 00000080:00010000:0.0:1787826360.200620:0:226485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801afb from it ffffb512c848bbe8 00010000:00010000:0.0:1787826360.200623:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826360.200631:0:226485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826360.200640:0:226485:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826360.200647:0:226485:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:0.0:1787826360.200681:0:226485:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826360.200687:0:226485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826360.200695:0:226485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826360.200701:0:226485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826360.200709:0:226485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826360.200715:0:226485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826360.200720:0:226485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826360.200722:0:226485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801afb 00000080:00010000:0.0:1787826360.200726:0:226485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801afb from it ffffb512c848bcb8 00010000:00010000:0.0:1787826360.200730:0:226485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826360.200737:0:226485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801afb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c2f7 expref: -99 pid: 226485 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826360.223322:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826360.268731:0:226474:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826360.268767:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826360.268839:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.268850:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.268861:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826360.268888:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681600:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826360.268999:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000100:00100000:3.0:1787826360.273720:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681600:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826360.273756:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273770:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273782:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273793:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273812:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273819:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273826:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826360.273839:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221880 x1874663332681600/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826360.273852:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273866:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826360.273874:0:226474:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826360.273911:0:226474:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826360.273918:0:226474:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801b10 00000080:00010000:3.0:1787826360.273924:0:226474:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826360.273927:0:226474:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b10 from it ffffb512c84cbbf8 00010000:00010000:3.0:1787826360.273930:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.273938:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826360.273983:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826360.274009:0:226474:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:3.0:1787826360.274014:0:226474:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826360.274038:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826360.274190:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.274199:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.274209:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826360.274267:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681728:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826360.274358:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000100:00100000:3.0:1787826360.309738:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681728:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826360.309773:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309799:0:226474:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309809:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309821:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309833:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309848:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309857:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309864:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826360.309877:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb221880 x1874663332681728/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826360.309890:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309902:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7772 (0 0) 00000002:00010000:3.0:1787826360.309922:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e5c:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826360.309982:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.309990:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.310000:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826360.310026:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681856:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826360.310127:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000100:00100000:3.0:1787826360.315278:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681856:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826360.315312:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315326:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315338:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315360:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315376:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315384:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315392:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826360.315404:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb220e00 x1874663332681856/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826360.315422:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315437:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7772 (0 0) 00000080:00200000:3.0:1787826360.315459:0:226474:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:3.0:1787826360.315579:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826360.315633:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315641:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826360.315649:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826360.315682:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681984:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826360.315775:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d600 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.319065:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332681984:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826360.319095:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319110:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319124:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319137:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319159:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319169:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319177:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.319194:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221880 x1874663332681984/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826360.319208:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319249:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677440/412 (0 0) 00010000:00010000:0.0:1787826360.319259:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319270:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826360.319305:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826360.319374:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319381:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.319389:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.319411:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682112:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826360.319504:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.338893:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682112:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826360.338928:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.338944:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.338959:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.338972:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.338986:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.338995:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339003:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.339017:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158df80 x1874663332682112/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826360.339031:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339046:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935617/410 (0 0) 00010000:00010000:0.0:1787826360.339054:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339068:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.339113:0:226474:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5c:0x0], updating i_size 8192 00000080:00200000:0.0:1787826360.339118:0:226474:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e5c:0x0](ffff9057dfe1ba88): rc = 0 00000080:00010000:0.0:1787826360.339129:0:226474:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1ba88) for remote lock 0x95194e8760801b1e 00000080:00010000:0.0:1787826360.339135:0:226474:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1ba88) for lock 0x95194e8760801b17 00000080:00010000:0.0:1787826360.339160:0:226474:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1ba88) 00000080:00010000:0.0:1787826360.339165:0:226474:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b17 from it ffffb512c84cbac8 00010000:00010000:0.0:1787826360.339169:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339176:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826360.339183:0:226474:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801b1e from it ffffb512c84cbac8 00010000:00010000:0.0:1787826360.339186:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339194:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.339267:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826360.339294:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826360.339300:0:226474:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), file ffff9057d887b080, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826360.339310:0:226474:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), intent=open|creat 00000002:00010000:0.0:1787826360.339342:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x200000403:0x1dc2:0x0]) in obj [0x200000401:0x19a:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826360.339365:0:226474:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.339640:0:226474:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 1/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.339648:0:226474:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826360.339652:0:226474:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3200/0x95194e8760801b2c lrc: 0/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c3a6 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.339705:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x19a:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.339713:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826360.339724:0:226474:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826360.339731:0:226474:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158df80 x1874663332682368/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826360.339740:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/0,1 mode: --/CW res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.339764:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682368:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826360.339935:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b510c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.348393:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682368:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826360.348438:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/0,1 mode: --/CW res: [0x200000401:0x19a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348481:0:226474:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348501:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/0,1 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348514:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348527:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/0,1 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348542:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348552:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 5/1,1 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348560:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.348574:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158df80 x1874663332682368/t8589957481(8589957481) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826360.348588:0:226474:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158df80 x1874663332682368/t8589957481(8589957481) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826360.348603:0:226474:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826360.348618:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348630:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7618 (0 0) 00000080:00200000:0.0:1787826360.348636:0:226474:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000401:0x19a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826360.348653:0:226474:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), intent=open|creat 00000080:00200000:0.0:1787826360.348699:0:226474:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000403:0x1dc2:0x0](ffff9057dfe17448) 00000080:00200000:0.0:1787826360.348799:0:226474:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dc2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826360.348804:0:226474:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000403:0x1dc2:0x0](ffff9057dfe17448) 00000080:00010000:0.0:1787826360.348808:0:226474:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc2:0x0](ffff9057dfe17448) for lock 0x95194e8760801b33 00000080:00010000:0.0:1787826360.348818:0:226474:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dc2:0x0](ffff9057dfe17448) 00000080:00200008:0.0:1787826360.348836:0:226474:0:(file.c:1090:ll_file_open()) START file f45j.sanityn:[0x200000403:0x1dc2:0x0](ffff9057dfe17448), flags 101101 00000002:00100000:0.0:1787826360.348859:0:226474:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158df80 x1874663332682368/t8589957481(8589957481) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826360.348915:0:226474:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b33 from it ffff9057f9b5c4e0 00010000:00010000:0.0:1787826360.348921:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.348931:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.348955:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000403:0x1dc2:0x0](ffff9057dfe17448), inode mode 81a4 mask 40 00000080:00200008:0.0:1787826360.349054:0:226474:0:(file.c:2650:do_file_write_iter()) START file f45j.sanityn:[0x200000403:0x1dc2:0x0], ppos: 0, count: 6 00000080:00200000:0.0:1787826360.349069:0:226474:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826360.349076:0:226474:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 6 00000080:00200000:0.0:1787826360.349081:0:226474:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: write ppos: 0, bytes: 6 00000080:00200000:0.0:1787826360.349087:0:226474:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826360.349096:0:226474:0:(cl_io.c:214:cl_io_rw_init()) header@000000008cf2c840[0x110, 1, [0x200000403:0x1dc2:0x0] hash] 00000020:00200000:0.0:1787826360.349099:0:226474:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6) 0 0 00000080:00200000:0.0:1787826360.349106:0:226474:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826360.349121:0:226474:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787826360.349130:0:226474:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6) 6, 6 00020000:00200000:0.0:1787826360.349134:0:226474:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826360.349165:0:226474:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787826360.349181:0:226474:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826360.349186:0:226474:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5dc30, osc lock ffff9057c8b83c80, flags 0x0 00010000:00010000:0.0:1787826360.349307:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.349318:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.349329:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 3/0,1 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.349354:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682496:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826360.349451:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b513c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.353177:0:226474:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973861 gen 19 00000100:00100000:0.0:1787826360.353563:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682496:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826360.353594:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,1 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.353616:0:226474:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.353635:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.353651:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826360.353667:0:226474:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.353744:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.353753:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826360.353778:0:226474:0:(vvp_io.c:1292:vvp_io_write_start()) f45j.sanityn: write [0, 6) append 0 append lockpos 0 00000080:00200000:0.0:1787826360.353801:0:226474:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 6 bytes 00000080:00200000:0.0:1787826360.353869:0:226474:0:(rw26.c:871:ll_write_end()) pos 0, len 6, copied 6 00000080:00200000:0.0:1787826360.354877:0:226474:0:(rw26.c:929:ll_write_end()) page@000000005f5c6867[3 00000000d5bc86ed 2 1 000000008b3599bf 0 0 0] 00000080:00200000:0.0:1787826360.354883:0:226474:0:(rw26.c:929:ll_write_end()) vmpage @00000000026ab0f5 17ffffc0001001 3:0 ffff9057fb2baee0 0 lru 00000080:00200000:0.0:1787826360.355058:0:226474:0:(rw26.c:929:ll_write_end()) osc-page@000000000e2a7a8d 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 0000000072dd0513 000000007dff1c01 > 3< 0 0 > 4< 0 0 8 484638720 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787826360.355062:0:226474:0:(rw26.c:929:ll_write_end()) end page@000000005f5c6867 00000080:00200000:0.0:1787826360.355063:0:226474:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787826360.355070:0:226474:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 6 prio 0 00000008:00010000:0.0:1787826360.355103:0:226474:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005746e05d ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826360.355136:0:226474:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff9057dfe175f8, count 1, dirtied 1 00000080:00200000:0.0:1787826360.355143:0:226474:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 6 bytes, tot: 6 00000080:00200000:0.0:1787826360.355147:0:226474:0:(vvp_io.c:1358:vvp_io_write_start()) f45j.sanityn: write bytes 0, result: 6 00000020:00010000:0.0:1787826360.355177:0:226474:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826360.355181:0:226474:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826360.355184:0:226474:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5dc30 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826360.355191:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.355202:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826360.355240:0:226474:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2128 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826360.355244:0:226474:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5dc30 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826360.355255:0:226474:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826360.355263:0:226474:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787826360.355274:0:226474:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826360.355277:0:226474:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 2 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826360.355287:0:226474:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc2:0x0] updating i_size 6 i_blocks 0 00000080:00200000:0.0:1787826360.355291:0:226474:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 6 00000080:00200008:0.0:1787826360.355335:0:226474:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dc2:0x0](ffff9057dfe17448), flags 100001 00000002:00100000:0.0:1787826360.356062:0:226474:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057f158df80 x1874663332682368/t8589957481(8589957481) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826376 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826360.356082:0:226474:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826360.356090:0:226474:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158dc00 x1874663332682752/t0(0) o35->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826360.356126:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682752:192.168.203.156@tcp:35:bash.0 00000100:00100000:0.0:1787826360.356823:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b513c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.368158:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332682752:192.168.203.156@tcp:35:bash.0 00000002:00100000:0.0:1787826360.368209:0:226474:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057f158df80 x1874663332682368/t8589957481(8589957481) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826376 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826360.368649:0:226474:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826360.368680:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826360.368702:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.368719:0:226474:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.368727:0:226474:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826360.368733:0:226474:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826360.368737:0:226474:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801b10 00000080:00010000:0.0:1787826360.368745:0:226474:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b10 from it ffffb512c84cbbf8 00010000:00010000:0.0:1787826360.368749:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.368758:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826360.368774:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826360.368781:0:226474:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=25 00000080:00200000:0.0:1787826360.368788:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826360.368803:0:226474:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826360.368808:0:226474:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), file ffff9057c6954d80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1787826360.368816:0:226474:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), intent=open|creat 00000002:00010000:0.0:1787826360.368841:0:226474:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x240000403:0x1e5d:0x0]) in obj [0x240000400:0x19c:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1787826360.368856:0:226474:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.368919:0:226474:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.368928:0:226474:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826360.368931:0:226474:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801b25 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c398 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.368972:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19c:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.368980:0:226474:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826360.368993:0:226474:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826360.369002:0:226474:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158e680 x1874663332683008/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826360.369012:0:226474:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/0,1 mode: --/CW res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226474 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826360.369035:0:226474:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332683008:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826360.369209:0:226474:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b513c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.378107:0:226474:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226474:1874663332683008:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826360.378146:0:226474:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/0,1 mode: --/CW res: [0x240000400:0x19c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378195:0:226474:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378205:0:226474:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/0,1 mode: --/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378272:0:226474:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378286:0:226474:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/0,1 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378303:0:226474:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378312:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 5/1,1 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378320:0:226474:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826360.378334:0:226474:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158e680 x1874663332683008/t4294985482(4294985482) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826360.378349:0:226474:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158e680 x1874663332683008/t4294985482(4294985482) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826360.378365:0:226474:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826360.378379:0:226474:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.378391:0:226474:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7773 (0 0) 00000080:00200000:0.0:1787826360.378398:0:226474:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240000400:0x19c:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826360.378813:0:226474:0:(namei.c:1628:ll_create_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e5c:0x0](ffff9057dfe1ba88), intent=open|creat 00000080:00200000:0.0:1787826360.378851:0:226474:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e5d:0x0](ffff9057dfe14b08) 00000080:00200000:0.0:1787826360.379145:0:226474:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e5d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826360.379163:0:226474:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240000403:0x1e5d:0x0](ffff9057dfe14b08) 00000080:00010000:0.0:1787826360.379167:0:226474:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5d:0x0](ffff9057dfe14b08) for lock 0x95194e8760801b41 00000080:00010000:0.0:1787826360.379179:0:226474:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5d:0x0](ffff9057dfe14b08) 00000080:00200008:0.0:1787826360.379196:0:226474:0:(file.c:1090:ll_file_open()) START file f45j.sanityn-2:[0x240000403:0x1e5d:0x0](ffff9057dfe14b08), flags 101101 00000002:00100000:0.0:1787826360.379381:0:226474:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158e680 x1874663332683008/t4294985482(4294985482) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826376 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826360.379787:0:226474:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b41 from it ffff9057ca869b40 00010000:00010000:0.0:1787826360.379795:0:226474:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826360.379807:0:226474:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826360.438787:0:226489:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826360.438811:0:226489:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787826360.438845:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826360.438851:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826360.438868:0:15673:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc1:0x0]: losing layout lock 00010000:00010000:3.0:1787826360.438907:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 1/0,0 mode: --/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826360.438914:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1787826360.438925:0:15673:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:398e0287-d3c7-44dd-b1d9-6b85f698e615:15673:1874663332683648:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826360.439767:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826360.439775:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826360.439796:0:6167:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e5d:0x0]: losing layout lock 00010000:00010000:0.0:1787826360.439830:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.439841:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801aa0 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c145 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.439867:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.439875:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.439880:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801aa0 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c145 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.439901:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1787826360.439916:0:6167:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:fc0b407b-a541-420d-8712-b9c8da18fe8c:6167:1874663332683776:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826360.440590:0:15673:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5d3c0 going to sleep for 11 seconds 00010000:00010000:3.0:1787826360.440880:0:226490:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00000100:00100000:0.0:1787826360.449265:0:6167:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826360.460986:0:6167:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:fc0b407b-a541-420d-8712-b9c8da18fe8c:6167:1874663332683776:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826360.461024:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d2400/0x95194e8760801b41 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c3ec expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.461063:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f8200/0x95194e8760801b1e lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c38a expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.461079:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057f9b07000/0x95194e8760801aa0 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5a:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c145 expref: -99 pid: 226437 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.461105:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826360.461107:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826360.461110:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826360.461135:0:6166:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239880 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:6166:1874663332683520:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826360.461159:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801b10 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c360 expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826360.461175:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801b33 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c3c2 expref: -99 pid: 226474 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.461189:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f8c00/0x95194e8760801b17 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c37c expref: -99 pid: 226474 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826360.461212:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826360.461214:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826360.461216:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826360.461381:0:15673:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:398e0287-d3c7-44dd-b1d9-6b85f698e615:15673:1874663332683648:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826360.461397:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801ac3 lrc: 0/0,0 mode: --/CR res: [0x200000403:0x1dc1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c1d8 expref: -99 pid: 226435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826360.461432:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826360.461434:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826360.461435:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826361.125744:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826362.146417:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826363.174757:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826364.090872:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346335488 00000100:00100000:3.0:1787826364.090886:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346335488 00000100:00100000:3.0:1787826364.091007:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43681 00000100:00100000:3.0:1787826364.091021:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8445:x1874663346335488:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826364.091044:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826364.091172:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826364.091180:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8445:x1874663346335488:12345-192.168.203.156@tcp:104: Request processed in 162us (667us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826364.091189:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43681 00010000:00010000:0.0:1787826364.091274:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826364.091284:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826364.091296:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2dc00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826364.091303:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400010000 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826364.091334:0:34881:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826364.091428:0:34881:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826364.094414:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826364.094438:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826364.094484:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826364.094494:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057c7539000 lock: 000000008a5ab064/0x95194e8760801a61 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x4d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c0a4 expref: -99 pid: 226254 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826364.094527:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826364.094530:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826364.094532:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826364.197212:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826364.502933:0:226655:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826364.502990:0:226655:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826364.503070:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826364.504622:0:226655:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826364.504709:0:226655:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226655 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826364.504750:0:226655:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226655:1874663332684032:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826364.504893:0:226655:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9ae93c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826364.835207:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332685056:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826364.835379:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332685312:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826364.835410:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332685568:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826364.835448:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1787826364.835455:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332684928:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826364.835484:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332685184:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826364.835507:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332685440:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826365.117934:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826365.118554:0:6634:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826365.118563:0:6634:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118572:0:6634:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826365.118578:0:6634:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118587:0:6634:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118596:0:6634:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118670:0:6634:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118679:0:6634:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826365.118706:0:6634:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118714:0:6634:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801b64 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c447 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.118738:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.118740:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.118742:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826365.118751:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826365.118781:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332686720:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00080000:3.0:1787826365.131279:0:2396:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9057d0de2680 x1874663332686976/t0(0) o101->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826365.137643:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332686720:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826365.149841:0:22705:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826365.149882:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346337664 00000100:00100000:3.0:1787826365.149897:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346337664 00000100:00100000:3.0:1787826365.149979:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43683 00000100:00100000:3.0:1787826365.149993:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c5062050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346337664:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826365.150018:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826365.150148:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826365.150157:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c5062050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6541:x1874663346337664:12345-192.168.203.156@tcp:104: Request processed in 166us (4117us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826365.150165:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43683 00010000:00010000:0.0:1787826365.150795:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826365.150806:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150824:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826365.150831:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150840:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150850:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150930:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 2/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150938:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826365.150964:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150972:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801b6b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c455 expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.150998:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.151000:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.151002:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826365.151022:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332687104:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826365.170014:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826365.171733:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826365.174418:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826365.174429:0:6167:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174441:0:6167:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2c800 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826365.174448:0:6167:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174459:0:6167:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174469:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826365.174497:0:6167:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc2:0x0]: losing layout lock 00010000:00010000:0.0:1787826365.174552:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 2/0,0 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174559:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826365.174599:0:6167:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174608:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801b72 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4c4e8 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826365.174643:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.174645:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.174647:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826365.174656:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826365.174674:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332687232:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826365.181532:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332687232:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787826365.183336:0:2396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183363:0:2396:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183397:0:2396:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183406:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183415:0:2396:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 5/1,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787826365.183432:0:2396:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 6, setting rss=6; leaving kms=0, end=18446744073709551615 00000020:00200000:3.0:1787826365.183441:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787826365.183459:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 4/1,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183467:0:2396:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 3/0,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183476:0:2396:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 4/0,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183484:0:2396:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06c00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:3.0:1787826365.183490:0:2396:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 5/0,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183503:0:2396:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1787826365.183512:0:2396:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 3/0,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826365.183520:0:2396:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000002a809666/0x95194e8760801b9c lrc: 0/0,0 mode: --/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826365.183550:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:0:1874663332686976:192.168.203.156@tcp:101:cat.0 00000100:00100000:3.0:1787826365.211463:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826365.212464:0:6166:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826365.212474:0:6166:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.212487:0:6166:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3200 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826365.212495:0:6166:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826365.212529:0:6166:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005746e05d ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826365.212545:0:6166:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005746e05d ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.212608:0:6166:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 5/0,0 mode: PW/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:0.0:1787826365.212692:0:6166:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158fb80 - object 0x280000401:1315 offset 0<>6 00000080:00200000:0.0:1787826365.218293:0:6166:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826365.218450:0:6166:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826365.218508:0:6166:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 2/0,0 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.218517:0:6166:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826365.219711:0:6166:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 1/0,0 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.219729:0:6166:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801b3a lrc: 0/0,0 mode: --/PW res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c3d0 expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.219762:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.219765:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826365.219767:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826365.220126:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826365.223015:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826365.223561:0:226655:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223573:0:226655:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223592:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826365.223607:0:226655:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223622:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223631:0:226655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826365.223663:0:226655:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826365.223691:0:226655:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc2:0x0] 00000008:00010000:0.0:1787826365.223701:0:226655:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826365.223705:0:226655:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d830, osc lock ffff9057c8b83c80, flags 0x400001002 00010000:00010000:0.0:1787826365.223726:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223739:0:226655:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826365.223749:0:226655:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826365.223752:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.223759:0:226655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826365.223764:0:226655:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826365.223771:0:226655:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc2:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826365.223775:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826365.223778:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.223780:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d830 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.223784:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826365.223787:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d830 (ffff9057d889c6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826365.223796:0:226655:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1787826365.223852:0:226655:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff9057c644bf00 00000008:00010000:0.0:1787826365.223901:0:215335:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9057c644bf00 00010000:00010000:0.0:1787826365.223926:0:215335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00100000:0.0:1787826365.223978:0:215335:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158ca80 - object 0x280000401:1315 offset 0<>4096 00000020:00010000:0.0:1787826365.233858:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826365.233864:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.233869:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057edd1eb30 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826365.233882:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826365.233901:0:226655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826365.233920:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826365.233923:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057edd1eb30 (ffff9057d889c6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826365.233933:0:226655:0:(vvp_io.c:533:vvp_io_advance()) advancing 6 bytes 00000080:00200000:0.0:1787826365.233947:0:226655:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826365.233953:0:226655:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 6, restart: 0 00000080:00200000:0.0:1787826365.233960:0:226655:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 6 00000080:00200008:0.0:1787826365.233986:0:226655:0:(file.c:2483:do_file_read_iter()) START file f45j.sanityn:[0x200000403:0x1dc2:0x0], ppos: 6, count: 4194304 00000080:00200000:0.0:1787826365.233997:0:226655:0:(file.c:2090:ll_file_io_generic()) f45j.sanityn: read ppos: 6, bytes: 4194304 00000080:00200000:0.0:1787826365.234002:0:226655:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826365.234009:0:226655:0:(cl_io.c:214:cl_io_rw_init()) header@00000000a860bd4c[0x110, 2, [0x200000403:0x1dc2:0x0] hash] 00000020:00200000:0.0:1787826365.234012:0:226655:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [6, 4194310) 0 0 00000080:00200000:0.0:1787826365.234016:0:226655:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826365.234034:0:226655:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 6 chunk: [6, 4194310) 4194310, 4194304 00020000:00200000:0.0:1787826365.234038:0:226655:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826365.234054:0:226655:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:0.0:1787826365.234068:0:226655:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826365.234073:0:226655:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d830, osc lock ffff9057c8b83c80, flags 0x0 00010000:00010000:0.0:1787826365.234097:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234114:0:226655:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234145:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234156:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234164:0:226655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826365.234173:0:226655:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [6, 4194310) 00000001:00010000:0.0:1787826365.234191:0:226655:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc2:0x0] 00000008:00010000:0.0:1787826365.234197:0:226655:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826365.234199:0:226655:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5ddb0, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826365.234206:0:226655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234215:0:226655:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826365.234266:0:226655:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826365.234270:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234287:0:226655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826365.234295:0:226655:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826365.234384:0:226655:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc2:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826365.234388:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826365.234391:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.234394:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5ddb0 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.234398:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f8a500 (ffff9057d889c6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826365.234401:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5ddb0 (ffff9057d889c6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826365.234408:0:226655:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:6 00000020:00010000:0.0:1787826365.234414:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826365.234417:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826365.234420:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d830 (ffff9057d889c3f8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826365.234425:0:226655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826365.234436:0:226655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 0000000005272d54/0x95194e8760801baa lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x523:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c54a expref: -99 pid: 226655 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:0.0:1787826365.234446:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da2998 (ffff9057d889c6c8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826365.234449:0:226655:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d830 (ffff9057d889c6c8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826365.234453:0:226655:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826365.234471:0:226655:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826365.234475:0:226655:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:0.0:1787826365.234479:0:226655:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:0.0:1787826365.234585:0:226655:0:(file.c:479:ll_file_release()) START file f45j.sanityn:[0x200000403:0x1dc2:0x0](ffff9057dfdd42c8), flags 100000 00000002:00100000:0.0:1787826365.234814:0:226655:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb239880 x1874663332684800/t8589957486(8589957486) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826380 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826365.234829:0:226655:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826365.234836:0:226655:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f800 x1874663332688128/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826365.234861:0:226655:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226655:1874663332688128:192.168.203.156@tcp:35:cat.0 00000100:00100000:0.0:1787826365.234961:0:226655:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826365.245612:0:226655:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226655:1874663332688128:192.168.203.156@tcp:35:cat.0 00000002:00100000:0.0:1787826365.245672:0:226655:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb239880 x1874663332684800/t8589957486(8589957486) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826380 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787826365.440593:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332688256:192.168.203.156@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1787826365.858632:0:160397:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826365.858657:0:160397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826365.858692:0:160397:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826365.858712:0:160397:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826365.858717:0:160397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787826365.858741:0:160397:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000074ff2f53 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e8760801b48 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c3fa expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1787826365.858767:0:160397:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000074ff2f53 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e8760801b48 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c3fa expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826365.858855:0:160397:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e8760801b48 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c3fa expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:3.0:1787826365.858968:0:160397:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057fb222300 - object 0x2c0000400:1230 offset 0<>6 00000100:00100000:3.0:1787826365.859062:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332688512:192.168.203.156@tcp:16:kworker.0 00000100:00100000:3.0:1787826365.859190:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787826365.859197:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332688384:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826365.870041:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332688512:192.168.203.156@tcp:16:kworker.0 00010000:00010000:0.0:1787826366.242530:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000002:00010000:3.0:1787826366.429833:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826366.429850:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826366.429864:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826366.429872:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826366.429881:0:226693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826366.429885:0:226693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801b4f 00000080:00010000:3.0:1787826366.429891:0:226693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b4f from it ffffb512c8663bf8 00010000:00010000:3.0:1787826366.429905:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826366.429913:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1787826366.429924:0:226693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826366.429933:0:226693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:3.0:1787826366.429943:0:226693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), inode mode 41ed mask 44 00000080:00200008:3.0:1787826366.429953:0:226693:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), flags 704400 00000002:00010000:3.0:1787826366.429972:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1787826366.430039:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826366.430048:0:226693:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787826366.430062:0:226693:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787826366.430072:0:226693:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb220000 x1874663332689152/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787826366.430082:0:226693:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826366.430125:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689152:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826366.430426:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9822d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.433810:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689152:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826366.433840:0:226693:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433856:0:226693:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433863:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433870:0:226693:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433879:0:226693:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433886:0:226693:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f8400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.433891:0:226693:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433910:0:226693:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433918:0:226693:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433925:0:226693:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826366.433933:0:226693:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.433940:0:226693:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801bc6 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.433974:0:226693:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9057fb220000 x1874663332689152/t4294985486(4294985486) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826366.433988:0:226693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220000 x1874663332689152/t4294985486(4294985486) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826366.434006:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826366.434050:0:226693:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5c:0x0], updating i_size 4096 00000002:00100000:0.0:1787826366.434073:0:226693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057fb220000 x1874663332689152/t4294985486(4294985486) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826366.434455:0:226693:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826366.434464:0:226693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8),name=d45j.sanityn 00000002:00010000:0.0:1787826366.434489:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.434516:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434529:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826366.434539:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.434545:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434553:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434563:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434571:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826366.434586:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.434594:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434610:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434618:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434627:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826366.434636:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826366.434641:0:226693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) 00000080:00010000:0.0:1787826366.434645:0:226693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) for lock 0x95194e8760801bb1 00000080:00010000:0.0:1787826366.434649:0:226693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801bb1 from it ffffb512c8663cd8 00010000:00010000:0.0:1787826366.434653:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.434661:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826366.434670:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.434675:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434682:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434690:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434697:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826366.434704:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.434709:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434716:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434724:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.434730:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826366.434817:0:226693:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826366.434831:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826366.434853:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826366.434858:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.434865:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826366.435523:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689280:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826366.435655:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.444397:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689280:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826366.445788:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.445811:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000002:00010000:0.0:1787826366.445842:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826366.445861:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.445875:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787826366.448827:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689408:192.168.203.156@tcp:37:rm.0 00000100:00100000:0.0:1787826366.449128:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.467358:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689408:192.168.203.156@tcp:37:rm.0 00010000:00010000:0.0:1787826366.468176:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.468197:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826366.468432:0:226693:0:(file.c:479:ll_file_release()) START file d45j.sanityn:[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), flags 704000 00000002:00100000:0.0:1787826366.468534:0:226693:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057fb220000 x1874663332689152/t4294985486(4294985486) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826382 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787826366.468548:0:226693:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787826366.468554:0:226693:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f800 x1874663332689664/t0(0) o35->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787826366.468588:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689664:192.168.203.156@tcp:35:rm.0 00000100:00100000:0.0:1787826366.468706:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.497363:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689664:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826366.497427:0:226693:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb220000 x1874663332689152/t4294985486(4294985486) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826382 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826366.497532:0:226693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826366.497556:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.497727:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.497746:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826366.497755:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826366.497761:0:226693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826366.497765:0:226693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801b4f 00000080:00010000:0.0:1787826366.497774:0:226693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801b4f from it ffffb512c8663bf8 00010000:00010000:0.0:1787826366.497779:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.497789:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826366.498055:0:226693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826366.498068:0:226693:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826366.498078:0:226693:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), inode mode 41ed mask 44 00000080:00200008:0.0:1787826366.498089:0:226693:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), flags 704400 00000002:00010000:0.0:1787826366.498119:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1787826366.498493:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.498505:0:226693:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826366.498522:0:226693:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826366.498533:0:226693:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f800 x1874663332689920/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826366.498543:0:226693:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826366.498588:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689920:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826366.498993:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.510896:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332689920:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826366.510969:0:226693:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.510983:0:226693:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.510997:0:226693:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.511011:0:226693:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 4/1,0 mode: CR/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.511030:0:226693:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9057f158f800 x1874663332689920/t4294985488(4294985488) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826366.511044:0:226693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f800 x1874663332689920/t4294985488(4294985488) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 648/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826366.511060:0:226693:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.511072:0:226693:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 16 res 9663677443/7772 (0 0) 00000080:00200000:0.0:1787826366.511080:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826366.511137:0:226693:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5c:0x0], updating i_size 4096 00000080:00010000:0.0:1787826366.511146:0:226693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) for lock 0x95194e8760801bcd 00000080:00010000:0.0:1787826366.511155:0:226693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801bcd from it ffffb512c8663bd0 00010000:00010000:0.0:1787826366.511160:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 3/1,0 mode: CR/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.511171:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801bcd lrc: 2/0,0 mode: CR/CR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c59e expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1787826366.511205:0:226693:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f800 x1874663332689920/t4294985488(4294985488) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826382 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1787826366.511362:0:226693:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826366.511371:0:226693:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8),name=d45j.sanityn 00000002:00010000:0.0:1787826366.511396:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.511409:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.511422:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:0.0:1787826366.511432:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.511439:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511448:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511457:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511466:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000002:00010000:0.0:1787826366.511476:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.511485:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511493:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511502:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.511510:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826366.511520:0:226693:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826366.511526:0:226693:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) 00000080:00010000:0.0:1787826366.511530:0:226693:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) for lock 0x95194e8760801bb1 00000080:00010000:0.0:1787826366.511533:0:226693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801bb1 from it ffffb512c8663cd8 00010000:00010000:0.0:1787826366.511536:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.511545:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b07e00/0x95194e8760801bb1 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5c:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c566 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:0.0:1787826366.511558:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.511564:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511572:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511581:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511588:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826366.511596:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.511601:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511613:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511621:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.511628:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826366.511728:0:226693:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff9057d128a800) d45j.sanityn proj=0 00000080:00000004:0.0:1787826366.511734:0:226693:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9057d128a800: age 8175, max_age 8181 00800000:00000004:0.0:1787826366.511747:0:226693:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: age 8175, max_age 8181 00000100:00100000:0.0:1787826366.511791:0:226693:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332690048:192.168.203.156@tcp:41:rm.0 00000100:00100000:0.0:1787826366.511907:0:226693:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.524184:0:226693:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226693:1874663332690048:192.168.203.156@tcp:41:rm.0 00800000:00000004:0.0:1787826366.524268:0:226693:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9057d128a800: update ffff9057c508d178 cache blocks 1793371/1916558 objects 519515/534210 00000080:00000004:0.0:1787826366.524277:0:226693:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9057d128a800: update ffff9057c5088450 cache blocks 1793371/1916558 objects 519515/534210 00000080:00000004:0.0:1787826366.524283:0:226693:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1793371/1916558 objects 519515/534210 00000080:00200000:0.0:1787826366.524329:0:226693:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240000403:0x1e5c:0x0](ffff9057dfe1a1c8) pos/size0/8192 32bit_api 0 00010000:00010000:0.0:1787826366.524370:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00010000:0.0:1787826366.524437:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19c:0x0]) in obj [0x240000400:0x19c:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826366.524451:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826366.524469:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826366.524485:0:226693:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826366.524490:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826366.524499:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:0.0:1787826366.524519:0:226693:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19a:0x0]) in obj [0x200000401:0x19a:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826366.524527:0:226693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826366.524535:0:226693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000002:00200000:0.0:1787826366.524849:0:226693:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:0.0:1787826366.524854:0:226693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826366.524862:0:226693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000100:00100000:3.0:1787826366.560744:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826366.563762:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826366.563772:0:34880:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.563789:0:34880:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b07600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.563795:0:34880:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.563816:0:34880:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.563826:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.564005:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 2/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.564013:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.564851:0:34880:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 1/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.564866:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07600/0x95194e8760801bbf lrc: 0/0,0 mode: --/PR res: [0x200000401:0x19a:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c582 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.564903:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.564905:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.564908:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826366.588813:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332690944:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826366.613928:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332690944:192.168.203.156@tcp:103:rm.0 00000100:00100000:3.0:1787826366.629681:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826366.629751:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826366.630410:0:34881:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826366.630419:0:34881:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630428:0:34881:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f8e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.630434:0:34881:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630444:0:34881:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630453:0:34881:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630523:0:34881:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630530:0:34881:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.630562:0:34881:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630570:0:34881:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8e00/0x95194e8760801b4f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c40f expref: -99 pid: 226655 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1787826366.630595:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.630597:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.630599:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826366.630607:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826366.630625:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332691584:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826366.631369:0:49150:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826366.631385:0:49150:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631394:0:49150:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2dc00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.631398:0:49150:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631407:0:49150:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631414:0:49150:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631457:0:49150:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631464:0:49150:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.631478:0:49150:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631485:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2dc00/0x95194e8760801b79 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c478 expref: -99 pid: 226657 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.631504:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.631506:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.631508:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826366.631523:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332691712:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826366.640393:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:0.0:1787826366.640980:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332691712:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.640996:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332691584:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.641568:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332691840:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.656837:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332691840:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826366.657970:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826366.668968:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826366.668979:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.668992:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06400 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.668998:0:49149:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669007:0:49149:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669017:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669182:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669201:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.669280:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669288:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801bb8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19c:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c574 expref: -99 pid: 226693 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1787826366.669315:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.669317:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.669319:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826366.669329:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826366.669349:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332691968:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.672956:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332691968:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.860195:0:226705:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226705:1874663332692224:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826366.860715:0:226705:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860732:0:226705:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860745:0:226705:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860757:0:226705:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860773:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860782:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860797:0:226705:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.860882:0:226705:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de1c00 x1874663332692224/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826382 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826366.860898:0:226705:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.860914:0:226705:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826366.860923:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826366.861202:0:226705:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826366.861212:0:226705:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801be2 00000080:00010000:0.0:1787826366.861473:0:226705:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826366.861478:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801be2 from it ffffb512c8703b58 00010000:00010000:0.0:1787826366.861483:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.861494:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826366.861531:0:226705:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826366.861558:0:226705:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=13 00000080:00200000:0.0:1787826366.861564:0:226705:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:0.0:1787826366.861732:0:226705:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826366.861789:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.861869:0:226705:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.861885:0:226705:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826366.861914:0:226705:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226705:1874663332692352:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826366.862315:0:226705:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.874576:0:226705:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226705:1874663332692352:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826366.874604:0:226705:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874621:0:226705:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874638:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874647:0:226705:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874656:0:226705:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874662:0:226705:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.874667:0:226705:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874675:0:226705:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874684:0:226705:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874692:0:226705:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826366.874699:0:226705:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.874705:0:226705:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801be9 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.874725:0:226705:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9057f158f480 x1874663332692352/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826382 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826366.874740:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787826366.874774:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.874789:0:226705:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826366.874799:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801be2 from it ffffb512c8703930 00010000:00010000:0.0:1787826366.874803:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.874812:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826366.874909:0:226705:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826366.874925:0:226705:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.874932:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.874940:0:226705:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826366.874947:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826366.874951:0:226705:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826366.874955:0:226705:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801be2 00000080:00010000:0.0:1787826366.874959:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801be2 from it ffffb512c8703be8 00010000:00010000:0.0:1787826366.874963:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826366.874971:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826366.874978:0:226705:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826366.874986:0:226705:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=16392 00000080:00200008:0.0:1787826366.875009:0:226705:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826366.875014:0:226705:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826366.875019:0:226705:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.875024:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.875033:0:226705:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826366.875039:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826366.875042:0:226705:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826366.875045:0:226705:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801be2 00000080:00010000:0.0:1787826366.875048:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801be2 from it ffffb512c8703cb8 00010000:00010000:0.0:1787826366.875051:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826366.875058:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1787826366.875734:0:226705:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1787826366.875740:0:226705:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826366.875752:0:226705:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.875805:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.875814:0:226705:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.875824:0:226705:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226705 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826366.875846:0:226705:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226705:1874663332692480:192.168.203.156@tcp:101:lfs.0 00000100:00100000:0.0:1787826366.875947:0:226705:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.880994:0:226705:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:398e0287-d3c7-44dd-b1d9-6b85f698e615:226705:1874663332692480:192.168.203.156@tcp:101:lfs.0 00010000:00010000:0.0:1787826366.881029:0:226705:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881043:0:226705:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881063:0:226705:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881074:0:226705:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881089:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881098:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881121:0:226705:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.881134:0:226705:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f480 x1874663332692480/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826382 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826366.881149:0:226705:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881163:0:226705:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826366.881173:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826366.881208:0:226705:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826366.881215:0:226705:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801bf0 00000080:00010000:0.0:1787826366.881241:0:226705:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826366.881246:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801bf0 from it ffffb512c8703cb8 00010000:00010000:0.0:1787826366.881250:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.881259:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826366.881738:0:226705:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826366.881766:0:226705:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.881780:0:226705:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.881794:0:226705:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826366.881938:0:226705:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826366.881945:0:226705:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826366.881949:0:226705:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801be2 00000080:00010000:0.0:1787826366.881956:0:226705:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801be2 from it ffffb512c8703bb0 00010000:00010000:0.0:1787826366.881959:0:226705:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826366.881968:0:226705:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826366.881980:0:226705:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 44 00000080:00200008:0.0:1787826366.881991:0:226705:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200080:0.0:1787826366.882020:0:226705:0:(dir.c:2001:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) cmd=c03066f0 arg=7ffd56203d20 00000080:00200000:0.0:1787826366.882036:0:226705:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988) name=d45j.sanityn stripe_offset=1 stripe_count=2, hash_type=4 00000020:00100000:0.0:1787826366.882117:0:226705:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826366.882128:0:226705:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f480 x1874663332692736/t0(0) o36->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826366.882158:0:226705:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226705:1874663332692736:192.168.203.156@tcp:36:lfs.0 00000100:00100000:0.0:1787826366.884078:0:226705:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:3.0:1787826366.894346:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826366.894509:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826366.894548:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346348288 00000100:00100000:3.0:1787826366.894560:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346348288 00000100:00100000:3.0:1787826366.894760:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346348416 00000100:00100000:3.0:1787826366.894765:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346348416 00000100:00100000:3.0:1787826366.894789:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43693 00000100:00100000:3.0:1787826366.894872:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb222a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346348288:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826366.894898:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826366.894997:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826366.895004:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb222a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6533:x1874663346348288:12345-192.168.203.156@tcp:104: Request processed in 206us (685us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826366.895012:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43693 00000100:00100000:3.0:1787826366.895033:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43694 00000100:00100000:3.0:1787826366.895040:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb220380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346348416:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826366.895049:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1787826366.895080:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1787826366.895086:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb220380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:6533:x1874663346348416:12345-192.168.203.156@tcp:104: Request processed in 46us (577us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826366.895092:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43694 00010000:00010000:0.0:1787826366.896015:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826366.896027:0:6167:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896039:0:6167:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.896045:0:6167:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896055:0:6167:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896065:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896149:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896157:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.896185:0:6167:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896194:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801bf0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c67e expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826366.896244:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826366.896248:0:6167:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896256:0:6167:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826366.896260:0:6167:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896267:0:6167:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896275:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896315:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896335:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826366.896342:0:6167:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896349:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057f9b06c00/0x95194e8760801be2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c65b expref: -99 pid: 226705 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826366.896364:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.896366:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826366.896368:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826366.896399:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332692864:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.896496:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826366.896504:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332692992:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.904651:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332692864:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.907516:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332692992:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826366.924037:0:226705:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job lfs:fc0b407b-a541-420d-8712-b9c8da18fe8c:226705:1874663332692736:192.168.203.156@tcp:36:lfs.0 00000080:00200000:0.0:1787826366.924124:0:226705:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88) 00000080:00200000:0.0:1787826366.924133:0:226705:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5e:0x0], updating i_size 4096 00000080:00200008:0.0:1787826366.924182:0:226705:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9057c9199988), flags 300000 00000080:00200000:0.0:1787826366.930113:0:226694:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826366.930155:0:226694:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826366.931587:0:226694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.931607:0:226694:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.931631:0:226694:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826366.931668:0:226694:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226694:1874663332693120:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826366.931795:0:226694:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.935747:0:226694:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226694:1874663332693120:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826366.935786:0:226694:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935871:0:226694:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935885:0:226694:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935898:0:226694:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935914:0:226694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935923:0:226694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935931:0:226694:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826366.935944:0:226694:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d880 x1874663332693120/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826382 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826366.935969:0:226694:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.935985:0:226694:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826366.935993:0:226694:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826366.936033:0:226694:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826366.936042:0:226694:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801bf7 00000080:00010000:0.0:1787826366.936048:0:226694:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826366.936051:0:226694:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801bf7 from it ffffb512c86bbbf8 00010000:00010000:0.0:1787826366.936055:0:226694:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.936064:0:226694:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826366.936094:0:226694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826366.936129:0:226694:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=25 00000080:00200000:0.0:1787826366.936135:0:226694:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:0.0:1787826366.936163:0:226694:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826366.936214:0:226694:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.936257:0:226694:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826366.936269:0:226694:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226694 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826366.936295:0:226694:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226694:1874663332693248:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826366.936737:0:226694:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00000100:00100000:0.0:1787826366.949892:0:226694:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job bash:fc0b407b-a541-420d-8712-b9c8da18fe8c:226694:1874663332693248:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826366.949929:0:226694:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c6e0 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826367.059135:0:226708:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826367.061422:0:226707:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057c7539000 00010000:00010000:0.0:1787826367.062610:0:226708:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/7 locks to regular blp list, 3 blwis in pool 00010000:00010000:0.0:1787826367.065656:0:15672:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 4 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1787826367.065670:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826367.065727:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2da00/0x95194e8760801c1a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c726 expref: -99 pid: 226694 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826367.065746:0:15672:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc3:0x0]: losing layout lock 00010000:00010000:0.0:1787826367.065782:0:15672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c6e0 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826367.066217:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801bf7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c6c4 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826367.066258:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057edd2da00/0x95194e8760801c1a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c726 expref: -99 pid: 226694 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826367.066266:0:15672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2200/0x95194e8760801bfe lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c6e0 expref: -99 pid: 226694 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826367.066273:0:15672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:0.0:1787826367.066303:0:15672:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_04:fc0b407b-a541-420d-8712-b9c8da18fe8c:15672:1874663332695040:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826367.066557:0:15672:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25540 going to sleep for 11 seconds 00010000:00010000:0.0:1787826367.069599:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787826367.069630:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07200/0x95194e8760801ba3 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c543 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826367.069671:0:15673:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc2:0x0]: losing layout lock 00010000:00010000:0.0:1787826367.069743:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07200/0x95194e8760801ba3 lrc: 1/0,0 mode: --/CR res: [0x200000403:0x1dc2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c543 expref: -99 pid: 226655 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826367.069751:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826367.069781:0:15673:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:398e0287-d3c7-44dd-b1d9-6b85f698e615:15673:1874663332695168:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826367.069917:0:15673:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25480 going to sleep for 11 seconds 00010000:00010000:3.0:1787826367.071912:0:226707:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787826367.076381:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826367.268598:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826369.315469:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826369.743254:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346351360 00000100:00100000:3.0:1787826369.743274:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346351360 00000100:00100000:3.0:1787826369.743362:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43695 00000100:00100000:3.0:1787826369.743374:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35352:x1874663346351360:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826369.743402:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 000000000f5bcb78/0x95194e8760801b48 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x4ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4c3fa expref: -99 pid: 226474 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826369.743508:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826369.743515:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35352:x1874663346351360:12345-192.168.203.156@tcp:104: Request processed in 144us (483us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826369.743524:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43695 00000100:00100000:3.0:1787826369.967217:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332695680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967284:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332695936:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967299:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332696192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967310:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973863 gen 19 00000100:00100000:3.0:1787826369.967336:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332696448:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967347:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332695552:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967491:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057d128a800: committing for last_committed 4294985491 gen 1 00000100:00100000:3.0:1787826369.967507:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332695808:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967521:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332696064:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826369.967530:0:2396:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057c7539000: committing for last_committed 4294985495 gen 1 00000100:00100000:3.0:1787826369.967548:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332696320:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826370.339311:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826370.987137:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:3.0:1787826370.987927:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332696576:192.168.203.156@tcp:4:bash.0 00000100:00100000:3.0:1787826370.995937:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332696704:192.168.203.156@tcp:16:kworker.0 00000008:00010000:3.0:1787826370.996030:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000db338858 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787826370.996090:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000db338858 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787826370.996126:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332696576:192.168.203.156@tcp:4:bash.0 00000080:00200000:0.0:1787826370.996431:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826370.996458:0:192756:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826370.996470:0:192756:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787826370.996498:0:192756:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000eea3cff0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787826370.996514:0:192756:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000eea3cff0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826370.996573:0:192756:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1787826370.996670:0:192756:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158ea00 - object 0x280000400:1233 offset 0<>6 00000100:00100000:0.0:1787826370.997309:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332696832:192.168.203.156@tcp:4:bash.0 00000100:00100000:0.0:1787826370.997407:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826370.997415:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332696960:192.168.203.156@tcp:16:kworker.0 00000100:00100000:0.0:1787826371.004264:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332696960:192.168.203.156@tcp:16:kworker.0 00000080:00200000:0.0:1787826371.004365:0:192756:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e5f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1787826371.009298:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000eea3cff0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787826371.009349:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000eea3cff0 ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787826371.009387:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332696832:192.168.203.156@tcp:4:bash.0 00010000:00010000:0.0:1787826371.362524:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826372.295886:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697216:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826372.295923:0:226875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.295953:0:226875:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.295962:0:226875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.295976:0:226875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.295990:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296018:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296028:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296037:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.296050:0:226875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb238700 x1874663332697216/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.296062:0:226875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296083:0:226875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7774 (0 0) 00000002:00010000:0.0:1787826372.296105:0:226875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e5e:0x0], intent: lookup flags 0 00010000:00010000:0.0:1787826372.296161:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296169:0:226875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.296178:0:226875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.296205:0:226875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697344:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826372.296327:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d86a0180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.300654:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697344:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826372.300691:0:226875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300705:0:226875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300720:0:226875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300732:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300746:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300756:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300764:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.300776:0:226875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057f158d500 x1874663332697344/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.300791:0:226875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.300806:0:226875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7774 (0 0) 00000080:00200000:0.0:1787826372.300831:0:226875:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826372.301021:0:226875:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8) 00000002:00010000:0.0:1787826372.301057:0:226875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19d:0x0]) in obj [0x240000400:0x19d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826372.301129:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.301137:0:226875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.301147:0:226875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.301172:0:226875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697472:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826372.301305:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d86a0180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.309045:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697472:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826372.309092:0:226875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309108:0:226875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309121:0:226875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309133:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309148:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309164:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309173:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.309187:0:226875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332697472/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.309201:0:226875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309216:0:226875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/413 (0 0) 00010000:00010000:0.0:1787826372.309276:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309285:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826372.309321:0:226875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19b:0x0]) in obj [0x200000401:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826372.309379:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309386:0:226875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.309396:0:226875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.309423:0:226875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697600:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826372.309537:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d86a0180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.316778:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697600:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826372.316821:0:226875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.316836:0:226875:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.316850:0:226875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.316863:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319287:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319300:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319309:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.319329:0:226875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332697600/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.319493:0:226875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319510:0:226875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/411 (0 0) 00010000:00010000:0.0:1787826372.319586:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319595:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801c52 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7b9 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.319709:0:226875:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5e:0x0], updating i_size 8192 00000080:00010000:0.0:1787826372.319723:0:226875:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe121c8) for remote lock 0x95194e8760801c44 00000080:00010000:0.0:1787826372.319729:0:226875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe121c8) for lock 0x95194e8760801c3d 00000080:00010000:0.0:1787826372.319743:0:226875:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826372.319747:0:226875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c3d from it ffffb512c88ebac8 00010000:00010000:0.0:1787826372.319752:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319760:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2dc00/0x95194e8760801c3d lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c78f expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826372.319769:0:226875:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801c44 from it ffffb512c88ebac8 00010000:00010000:0.0:1787826372.319774:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.319783:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.319812:0:226875:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787826372.319843:0:226875:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), file ffff9057d06c7800, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1787826372.319849:0:226875:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), intent=open 00000002:00010000:0.0:1787826372.322290:0:226875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x19b:0x0], intent: open flags 0100001 00010000:00010000:0.0:1787826372.322396:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801c59 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.322409:0:226875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801c59 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826372.322424:0:226875:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826372.322440:0:226875:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d500 x1874663332697856/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826372.322451:0:226875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801c59 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.322477:0:226875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697856:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826372.324712:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d86a0180 going to sleep for 11 seconds 00000080:00200000:3.0:1787826372.796614:0:226877:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826372.796760:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826372.796837:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.796850:0:226877:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.796861:0:226877:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826372.796891:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332697984:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826372.797208:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9fd9c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826372.808679:0:226877:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957494 gen 1 00000100:00100000:3.0:1787826372.808729:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332697984:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826372.808830:0:226877:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808846:0:226877:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808860:0:226877:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808872:0:226877:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808887:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808897:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808905:0:226877:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826372.808917:0:226877:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb223100 x1874663332697984/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826372.808932:0:226877:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.808949:0:226877:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826372.809214:0:226877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826372.809463:0:226877:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826372.809473:0:226877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801c60 00000080:00010000:3.0:1787826372.809480:0:226877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826372.809484:0:226877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c60 from it ffffb512c8803bd0 00010000:00010000:3.0:1787826372.809489:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.809501:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826372.809597:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826372.809623:0:226877:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=24 00000080:00200000:3.0:1787826372.809629:0:226877:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d45j.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=0 00000002:00010000:3.0:1787826372.809657:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826372.809709:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.809718:0:226877:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.809728:0:226877:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826372.809822:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698112:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826372.809936:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9fd9c00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826372.827318:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698112:192.168.203.156@tcp:101:mrename.0 00010000:00010000:3.0:1787826372.827461:0:226877:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827494:0:226877:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827504:0:226877:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827517:0:226877:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827531:0:226877:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827547:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827557:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827565:0:226877:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826372.827580:0:226877:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb223100 x1874663332698112/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826372.827595:0:226877:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827608:0:226877:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 9663677443/7774 (0 0) 00000002:00010000:3.0:1787826372.827628:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e5e:0x0], intent: lookup flags 0 00010000:00010000:3.0:1787826372.827708:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827837:0:226877:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826372.827849:0:226877:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826372.827875:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698240:192.168.203.156@tcp:101:mrename.0 00000100:00100000:3.0:1787826372.828174:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9fd9c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.856399:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698240:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826372.856437:0:226877:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856454:0:226877:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856467:0:226877:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856479:0:226877:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856494:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856502:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856510:0:226877:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.856524:0:226877:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9057fb221880 x1874663332698240/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.856539:0:226877:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856556:0:226877:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7774 (0 0) 00000080:00200000:0.0:1787826372.856579:0:226877:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826372.856625:0:226877:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5e:0x0], updating i_size 4096 00000080:00200000:0.0:1787826372.856628:0:226877:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e5e:0x0](ffff9057dfe13a88): rc = 0 00000080:00010000:0.0:1787826372.856636:0:226877:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe13a88) for remote lock 0x95194e8760801c6e 00000080:00010000:0.0:1787826372.856641:0:226877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe13a88) for lock 0x95194e8760801c67 00000080:00010000:0.0:1787826372.856655:0:226877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe13a88) 00000080:00010000:0.0:1787826372.856659:0:226877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c67 from it ffffb512c8803aa0 00010000:00010000:0.0:1787826372.856663:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856674:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826372.856864:0:226877:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801c6e from it ffffb512c8803aa0 00010000:00010000:0.0:1787826372.856870:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.856878:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.856906:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826372.856935:0:226877:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826372.856955:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826372.857035:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826372.857048:0:226877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.857055:0:226877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826372.857073:0:226877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826372.857076:0:226877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801c60 00000080:00010000:0.0:1787826372.857080:0:226877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c60 from it ffffb512c8803bd0 00010000:00010000:0.0:1787826372.857084:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826372.857098:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826372.857108:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826372.857115:0:226877:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=24 00000080:00200000:0.0:1787826372.857121:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), inode mode 41ed mask 1 00000080:00200000:0.0:1787826372.857139:0:226877:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), flags=0 00000080:00200000:0.0:1787826372.857143:0:226877:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn-2, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), intent=getattr 00000002:00010000:0.0:1787826372.857160:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn-2,[0x0:0x0:0x0]) in obj [0x240000400:0x19d:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826372.857218:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.857343:0:226877:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.857355:0:226877:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.857383:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698368:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826372.857585:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edc7b9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.865930:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698368:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826372.865976:0:226877:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866016:0:226877:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866033:0:226877:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866047:0:226877:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866060:0:226877:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866086:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866097:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866106:0:226877:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.866120:0:226877:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332698368/t0(0) o101->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.866134:0:226877:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826372.866149:0:226877:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866168:0:226877:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c73cc400 type 13 mode 4 res 9663677443/7775 (0 0) 00000080:00200000:0.0:1787826372.866177:0:226877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x19d:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826372.866319:0:226877:0:(namei.c:147:ll_iget()) got inode: [0x240000403:0x1e5f:0x0](ffff9057dfe1e3c8): rc = 0 00000080:00200000:0.0:1787826372.866336:0:226877:0:(file.c:6843:ll_layout_conf()) [0x240000403:0x1e5f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826372.866342:0:226877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5f:0x0](ffff9057dfe1e3c8) for lock 0x95194e8760801c75 00000080:00010000:0.0:1787826372.866357:0:226877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5f:0x0](ffff9057dfe1e3c8) 00000080:00010000:0.0:1787826372.866362:0:226877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c75 from it ffffb512c8803db8 00010000:00010000:0.0:1787826372.866367:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866378:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.866436:0:226877:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), flags=2048 00000080:00200000:0.0:1787826372.866443:0:226877:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), intent=getattr 00000002:00010000:0.0:1787826372.866469:0:226877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f45j.sanityn,[0x0:0x0:0x0]) in obj [0x200000401:0x19b:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826372.866526:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866534:0:226877:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.866546:0:226877:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226877 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826372.866574:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698496:192.168.203.156@tcp:101:mrename.0 00000100:00100000:0.0:1787826372.866689:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edc7b9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.874416:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698496:192.168.203.156@tcp:101:mrename.0 00010000:00010000:0.0:1787826372.874451:0:226877:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874486:0:226877:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874494:0:226877:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: --/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874506:0:226877:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874518:0:226877:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874531:0:226877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874540:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 5/2,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874548:0:226877:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 4/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826372.874560:0:226877:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c700 x1874663332698496/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826388 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mrename.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826372.874573:0:226877:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826372.874600:0:226877:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874611:0:226877:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935619/7619 (0 0) 00000080:00200000:0.0:1787826372.874619:0:226877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x19b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826372.874679:0:226877:0:(namei.c:147:ll_iget()) got inode: [0x200000403:0x1dc3:0x0](ffff9057dfe1c2c8): rc = 0 00000080:00200000:0.0:1787826372.874704:0:226877:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dc3:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826372.874711:0:226877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc3:0x0](ffff9057dfe1c2c8) for lock 0x95194e8760801c7c 00000080:00010000:0.0:1787826372.874722:0:226877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000403:0x1dc3:0x0](ffff9057dfe1c2c8) 00000080:00010000:0.0:1787826372.874726:0:226877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c7c from it ffffb512c8803db8 00010000:00010000:0.0:1787826372.874731:0:226877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 3/1,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826372.874741:0:226877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.874779:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826372.874786:0:226877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), inode mode 41ed mask 3 00000080:00200000:0.0:1787826372.874796:0:226877:0:(namei.c:2492:ll_rename()) VFS Op:oldname=f45j.sanityn-2, src_dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88), newname=f45j.sanityn, tgt_dir=[0x240000403:0x1e5e:0x0](ffff9057dfe13a88) 00010000:00010000:0.0:1787826372.874834:0:226877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.874851:0:226877:0:(vvp_object.c:117:vvp_conf_set()) [0x240000403:0x1e5f:0x0]: losing layout lock 00010000:00010000:0.0:1787826372.874895:0:226877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826372.874907:0:226877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826372.874950:0:226877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9057d86d3e00/0x95194e8760801c75 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c822 expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826372.874988:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826372.875007:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332698624:192.168.203.156@tcp:103:mrename.0 00010000:00010000:0.0:1787826372.875134:0:226877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826372.875146:0:226877:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1dc3:0x0]: losing layout lock 00010000:00010000:0.0:1787826372.875170:0:226877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826372.875177:0:226877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826372.875180:0:226877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2600/0x95194e8760801c7c lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c83e expref: -99 pid: 226877 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826372.875209:0:226877:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826372.875216:0:226877:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f100 x1874663332698880/t0(0) o36->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:13/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826372.875286:0:226877:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698880:192.168.203.156@tcp:36:mrename.0 00000100:00100000:0.0:1787826372.875322:0:226877:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edc7b9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826372.881540:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332698624:192.168.203.156@tcp:103:mrename.0 00000100:00100000:3.0:1787826372.898029:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346355712 00000100:00100000:3.0:1787826372.898040:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346355712 00000100:00100000:3.0:1787826372.898123:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43696 00000100:00100000:3.0:1787826372.898136:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346355712:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826372.898161:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.898631:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826372.898641:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:125377:x1874663346355712:12345-192.168.203.156@tcp:104: Request processed in 507us (1028us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826372.898650:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43696 00010000:00010000:3.0:1787826372.898679:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787826372.898687:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.898715:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3600 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:3.0:1787826372.898721:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.898731:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.898932:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.899166:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.899175:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787826372.899202:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.899210:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801c4b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7ab expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826372.899264:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826372.899266:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826372.899269:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826372.899290:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332699008:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826372.902655:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332697856:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826372.902751:0:226875:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801c59 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c84c expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826372.917875:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332699008:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826372.947027:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346356352 00000100:00100000:3.0:1787826372.947050:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346356352 00000100:00100000:3.0:1787826372.947244:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43698 00000100:00100000:3.0:1787826372.947258:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35352:x1874663346356352:12345-192.168.203.156@tcp:106: 00010000:00010000:3.0:1787826372.947280:0:6158:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826372.947294:0:6158:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:3.0:1787826372.947326:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@000000001a8c538c[0x110, 2, [0x200000403:0x1dc3:0x0] hash] 00000020:00010000:3.0:1787826372.947328:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826366 atime: 1787826366 ctime: 1787826366 blocks: 8 00000020:00010000:3.0:1787826372.947331:0:6158:0:(cl_object.c:322:cl_object_glimpse()) header@000000003164e0ec[0x110, 2, [0x2c0000401:0x524:0x0] hash] 00000020:00010000:3.0:1787826372.947333:0:6158:0:(cl_object.c:322:cl_object_glimpse()) size: 6 mtime: 1787826366 atime: 1787826366 ctime: 1787826366 blocks: 8 00000100:00100000:3.0:1787826372.947419:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35352:x1874663346356352:12345-192.168.203.156@tcp:106: Request processed in 163us (3083us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826372.947426:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43698 00000100:00100000:0.0:1787826372.978415:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9ae8f00 going to sleep for 11 seconds 00000100:00100000:3.0:1787826372.980106:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346356992 00000100:00100000:3.0:1787826372.980124:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346356992 00000100:00100000:3.0:1787826372.980248:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43699 00000100:00100000:3.0:1787826372.980265:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346356992:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826372.980285:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2200/0x95194e8760801c59 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4c84c expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826372.980380:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826372.980387:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6540:x1874663346356992:12345-192.168.203.156@tcp:104: Request processed in 127us (3905us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826372.980395:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43699 00000100:00100000:3.0:1787826372.996190:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346357120 00000100:00100000:3.0:1787826372.996205:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346357120 00000100:00100000:3.0:1787826372.996319:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43700 00000100:00100000:3.0:1787826372.996334:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8445:x1874663346357120:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826372.996360:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826372.996467:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826372.996475:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8445:x1874663346357120:12345-192.168.203.156@tcp:104: Request processed in 145us (431us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826372.996483:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43700 00010000:00010000:0.0:1787826372.996752:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826372.996763:0:34880:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826372.996774:0:34880:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2d200 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826372.996782:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400000000 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826372.996815:0:34880:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826372.996912:0:34880:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826372.996975:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 2/0,0 mode: --/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826372.996984:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826372.997016:0:34880:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826372.997025:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057c7539000 lock: 0000000014db9e45/0x95194e8760801c21 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0xe21309aa81f4c734 expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826372.997054:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826372.997056:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826372.997059:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826372.997077:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826372.997096:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332699648:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826373.012409:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2d80 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332699392:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826373.012458:0:226875:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.012476:0:226875:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.012486:0:226875:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826373.015884:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346358528 00000100:00100000:3.0:1787826373.015897:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346358528 00000100:00100000:3.0:1787826373.015975:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43701 00000100:00100000:3.0:1787826373.015988:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6512:x1874663346358528:12345-192.168.203.156@tcp:105: 00010000:00010000:3.0:1787826373.016010:0:6159:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016025:0:6159:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->4194303] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016036:0:6159:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/1,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016050:0:6159:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016075:0:6159:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10080000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016086:0:6159:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x80000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826373.016130:0:6159:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9057f9b06200) 00000100:00100000:3.0:1787826373.016213:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6512:x1874663346358528:12345-192.168.203.156@tcp:105: Request processed in 227us (815us total) trans 0 rc 0/0 00010000:00010000:0.0:1787826373.016254:0:226875:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826373.016265:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43701 00010000:00010000:0.0:1787826373.016271:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x20000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016292:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10000000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826373.016309:0:226875:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 6, setting rss=6; kms=6, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x10020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016324:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016339:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787826373.016374:0:226875:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000020:00010000:0.0:1787826373.016390:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2da26c8 (ffff9057d889c5a8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826373.016395:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2da26c8 (ffff9057d889c5a8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826373.016400:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057edd1eb30 (ffff9057d889cb48/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787826373.016406:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016418:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787826373.016438:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2da26c8 (ffff9057d889c5a8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826373.016442:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057edd1eb30 (ffff9057d889c5a8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826373.016451:0:226875:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1787826373.016464:0:226875:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc3:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826373.016468:0:226875:0:(file.c:2238:ll_file_io_generic()) f45j.sanityn: 1 io complete with rc: 0, result: 0, restart: 1 00000080:00200000:0.0:1787826373.016472:0:226875:0:(file.c:2245:ll_file_io_generic()) f45j.sanityn: restart read from ppos=0 bytes=4194304 retries=999 ret=0: rc = 0 00000080:00200000:0.0:1787826373.016479:0:226875:0:(file.c:1848:ll_io_set_mirror()) f45j.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1787826373.016486:0:226875:0:(cl_io.c:214:cl_io_rw_init()) header@00000000f577748b[0x110, 1, [0x200000403:0x1dc3:0x0] hash] 00000020:00200000:0.0:1787826373.016488:0:226875:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1787826373.016494:0:226875:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc3:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826373.016527:0:226875:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000403:0x1dc3:0x0](ffff9057dfe19148) 00000002:00010000:0.0:1787826373.016549:0:226875:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000403:0x1dc3:0x0]) in obj [0x200000403:0x1dc3:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826373.016601:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016611:0:226875:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.016622:0:226875:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: --/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226875 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826373.016759:0:226875:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332699776:192.168.203.156@tcp:101:cat.0 00000100:00100000:0.0:1787826373.016836:0:226875:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d86a0180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826373.017730:0:226877:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job mrename:fc0b407b-a541-420d-8712-b9c8da18fe8c:226877:1874663332698880:192.168.203.156@tcp:36:mrename.0 00000100:00100000:0.0:1787826373.020532:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332699648:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826373.031617:0:226875:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job cat:398e0287-d3c7-44dd-b1d9-6b85f698e615:226875:1874663332699776:192.168.203.156@tcp:101:cat.0 00010000:00010000:0.0:1787826373.031671:0:226875:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.031690:0:226875:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 4/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826373.031706:0:226875:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158c380 x1874663332699776/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826389 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826373.031719:0:226875:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826373.031733:0:226875:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.031755:0:226875:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 16 res 8589935619/7619 (0 0) 00000080:00200000:0.0:1787826373.031763:0:226875:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000403:0x1dc3:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826373.031784:0:226875:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000403:0x1dc3:0x0](ffff9057dfe19148) for lock 0x95194e8760801c91 00000080:00010000:0.0:1787826373.031794:0:226875:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c91 from it ffffb512c88ebaf8 00010000:00010000:0.0:1787826373.031799:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826373.031810:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.031841:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.031856:0:226875:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 2/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826373.031866:0:226875:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000403:0x1dc3:0x0](000000002313416a) being reconfigured ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826373.031913:0:226875:0:(file.c:6843:ll_layout_conf()) [0x200000403:0x1dc3:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826373.031919:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 3/1,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.031927:0:226875:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2600/0x95194e8760801c91 lrc: 2/0,0 mode: CR/CR res: [0x200000403:0x1dc3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4c8a7 expref: -99 pid: 226875 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826373.031959:0:226875:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1787826373.031964:0:226875:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826373.031988:0:226875:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1787826373.032004:0:226875:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787826373.032009:0:226875:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5dab0, osc lock ffff9057c8b83c80, flags 0x0 00010000:00010000:0.0:1787826373.032028:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032038:0:226875:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032051:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032059:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032076:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826373.032086:0:226875:0:(vvp_io.c:835:vvp_io_read_start()) f45j.sanityn: read [0, 4194304) 00000001:00010000:0.0:1787826373.032105:0:226875:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000403:0x1dc3:0x0] 00000008:00010000:0.0:1787826373.032112:0:226875:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826373.032114:0:226875:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d430, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826373.032122:0:226875:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032135:0:226875:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826373.032144:0:226875:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826373.032148:0:226875:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826373.032155:0:226875:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x810020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826373.032162:0:226875:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826373.032169:0:226875:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000403:0x1dc3:0x0] updating i_size 6 i_blocks 8 00000020:00010000:0.0:1787826373.032173:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89970 (ffff9057d889c5a8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826373.032176:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89970 (ffff9057d889c5a8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826373.032180:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d430 (ffff9057d889cb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826373.032184:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89970 (ffff9057d889c5a8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826373.032187:0:226875:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d430 (ffff9057d889c5a8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826373.032196:0:226875:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1787826373.032309:0:226875:0:(osc_cache.c:3003:osc_queue_sync_pages()) ### no dlm lock: extent: ffff9057c644b3c0 00000008:00010000:0.0:1787826373.032381:0:215335:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff9057c644b3c0 00010000:00010000:0.0:1787826373.032407:0:215335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800020000020000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:0.0:1787826373.032453:0:215335:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9057f158c380 - object 0x2c0000401:1316 offset 0<>4096 00000100:00100000:0.0:1787826373.032509:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332699904:192.168.203.156@tcp:3:cat.0 00000008:00010000:0.0:1787826373.040308:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff9057c644b3c0 00000020:00200000:0.0:1787826373.040330:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1787826373.040349:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff9057c644b3c0 00000100:00100000:0.0:1787826373.040376:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332699904:192.168.203.156@tcp:3:cat.0 00010000:00010000:3.0:1787826373.410474:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.465255:0:226914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826374.465287:0:226914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826374.465304:0:226914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826374.465309:0:226914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801c36 00000080:00010000:0.0:1787826374.467128:0:226914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c36 from it ffffb512c895bb58 00010000:00010000:0.0:1787826374.467147:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.467165:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826374.467191:0:226914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826374.467208:0:226914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=8 00000080:00200008:0.0:1787826374.467244:0:226914:0:(file.c:6296:ll_getattr_dentry()) START file d45j.sanityn:[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826374.467251:0:226914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8),name=d45j.sanityn 00000002:00010000:0.0:1787826374.467292:0:226914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000403:0x1e5e:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826374.467307:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 30 res 9663677443/7774 (0 0) 00010000:00010000:0.0:1787826374.467374:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.467382:0:226914:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.467394:0:226914:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826374.467426:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700288:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826374.467574:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826374.470287:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700288:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826374.470318:0:226914:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470358:0:226914:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470366:0:226914:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470378:0:226914:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470390:0:226914:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470403:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470411:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470423:0:226914:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826374.470435:0:226914:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332700288/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/672 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826374.470448:0:226914:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470457:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7774 (0 0) 00000080:00200000:0.0:1787826374.470465:0:226914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826374.470503:0:226914:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000403:0x1e5e:0x0], updating i_size 4096 00000080:00010000:0.0:1787826374.470511:0:226914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe121c8) for lock 0x95194e8760801c98 00000080:00010000:0.0:1787826374.470516:0:226914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5e:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826374.470519:0:226914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c98 from it ffffb512c895bcb8 00010000:00010000:0.0:1787826374.470522:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470531:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826374.470568:0:226914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x19d:0x0]) in obj [0x240000400:0x19d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826374.470619:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470626:0:226914:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.470640:0:226914:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 3/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826374.470664:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700416:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826374.470760:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826374.477771:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700416:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826374.477807:0:226914:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 4/1,0 mode: --/CR res: [0x240000400:0x19d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477821:0:226914:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477836:0:226914:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477847:0:226914:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477862:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477870:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477877:0:226914:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826374.477895:0:226914:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332700416/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826374.477908:0:226914:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477922:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677440/413 (0 0) 00010000:00010000:0.0:1787826374.477930:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.477939:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3200/0x95194e8760801c9f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x19d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8ca expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:0.0:1787826374.477983:0:226914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x19b:0x0]) in obj [0x200000401:0x19b:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826374.478042:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.478059:0:226914:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.478070:0:226914:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826374.478095:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700544:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826374.478195:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826374.481069:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332700544:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826374.481100:0:226914:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481116:0:226914:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481130:0:226914:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481144:0:226914:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481159:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481168:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481176:0:226914:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826374.481187:0:226914:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332700544/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826374.481200:0:226914:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481215:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935617/411 (0 0) 00010000:00010000:0.0:1787826374.481255:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.481270:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826374.481379:0:226914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826374.481405:0:226914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826374.481418:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.481430:0:226914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826374.481437:0:226914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826374.481445:0:226914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826374.481448:0:226914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801c36 00000080:00010000:0.0:1787826374.481453:0:226914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801c36 from it ffffb512c895bbf8 00010000:00010000:0.0:1787826374.481456:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.481464:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801c36 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c773 expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826374.481474:0:226914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826374.481481:0:226914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d45j.sanityn, flags=266 00000080:00200000:0.0:1787826374.481695:0:226914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), inode mode 41ed mask 44 00000080:00200008:0.0:1787826374.481708:0:226914:0:(file.c:1090:ll_file_open()) START file d45j.sanityn:[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), flags 704400 00000100:00100000:0.0:1787826374.582897:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332701824:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826374.582932:0:226914:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: --/CR res: [0x200000401:0x19b:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.582959:0:226914:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.582968:0:226914:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.582981:0:226914:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.582993:0:226914:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583007:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583016:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583023:0:226914:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826374.583037:0:226914:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb238a80 x1874663332701824/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/584 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826374.583049:0:226914:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583076:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 9663677443/7775 (0 0) 00000002:00010000:0.0:1787826374.583098:0:226914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240000403:0x1e5f:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826374.583163:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583171:0:226914:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.583181:0:226914:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 3/1,0 mode: --/CR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 226914 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826374.583208:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332701952:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826374.583367:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826374.587182:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332701952:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826374.587216:0:226914:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 4/1,0 mode: --/CR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587274:0:226914:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 4/1,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587745:0:226914:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587759:0:226914:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587775:0:226914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587785:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 5/2,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587793:0:226914:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 4/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826374.587807:0:226914:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158c380 x1874663332701952/t0(0) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/640 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826374.587822:0:226914:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.587837:0:226914:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c55ed000 type 13 mode 4 res 9663677443/7775 (0 0) 00000080:00200000:0.0:1787826374.587859:0:226914:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x19b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826374.587903:0:226914:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000403:0x1e5f:0x0](ffff9057dfe10908) 00000080:00010000:0.0:1787826374.587980:0:226914:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5f:0x0](ffff9057dfe10908) for remote lock 0x95194e8760801cc2 00000080:00010000:0.0:1787826374.587987:0:226914:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000403:0x1e5f:0x0](ffff9057dfe10908) for lock 0x95194e8760801cbb 00000080:00010000:0.0:1787826374.587998:0:226914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240000403:0x1e5f:0x0](ffff9057dfe10908) 00000080:00010000:0.0:1787826374.588001:0:226914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cbb from it ffffb512c895bdf8 00010000:00010000:0.0:1787826374.588006:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.588016:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826374.588032:0:226914:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x95194e8760801cc2 from it ffffb512c895bdf8 00010000:00010000:0.0:1787826374.588036:0:226914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 3/1,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826374.588044:0:226914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826374.588077:0:226914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8), inode mode 41ed mask 3 00000080:00200000:0.0:1787826374.588088:0:226914:0:(namei.c:2418:ll_unlink()) VFS Op:name=f45j.sanityn, dir=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8) 00010000:00010000:0.0:1787826374.588131:0:226914:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.588184:0:226914:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.588191:0:226914:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826374.588257:0:226914:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2d400/0x95194e8760801cbb lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c917 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.591712:0:226914:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.591812:0:226914:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.591820:0:226914:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826374.591823:0:226914:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d2a00/0x95194e8760801cc2 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5f:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c925 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826374.591855:0:226914:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826374.591870:0:226914:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c700 x1874663332702336/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826374.591898:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332702336:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826374.592002:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b51a80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826374.594740:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346359424 00000100:00100000:3.0:1787826374.594753:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346359424 00000100:00100000:3.0:1787826374.594825:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43702 00000100:00100000:3.0:1787826374.594838:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346359424:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826374.594859:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3a00/0x95194e8760801ca6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x19b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c8d8 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1787826374.594959:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826374.594967:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:398e0287-d3c7-44dd-b1d9-6b85f698e615+5:6532:x1874663346359424:12345-192.168.203.156@tcp:104: Request processed in 132us (313us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826374.594975:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43702 00000100:00100000:0.0:1787826374.595329:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332702464:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826374.602451:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332702464:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826374.605809:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346360320 00000100:00100000:3.0:1787826374.605822:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346360320 00000100:00100000:3.0:1787826374.605935:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43703 00000100:00100000:3.0:1787826374.605948:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346360320:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826374.605985:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826374.606250:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826374.606258:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:17826:x1874663346360320:12345-192.168.203.156@tcp:104: Request processed in 312us (586us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826374.606265:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43703 00010000:00010000:0.0:1787826374.606340:0:49149:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826374.606349:0:49149:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800420000030000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.606359:0:49149:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057f9b06200 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826374.606365:0:49149:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x800428400030000 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826374.606391:0:49149:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826374.606465:0:49149:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826374.606519:0:49149:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.606527:0:49149:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826374.606556:0:49149:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.606564:0:49149:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 00000000a2593a80/0x95194e8760801c8a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4194303) gid 0 flags: 0x804c69400030020 nid: local remote: 0xe21309aa81f4c88b expref: -99 pid: 226875 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.606596:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.606598:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.606600:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826374.606609:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826374.606625:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332702592:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826374.610396:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332702592:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826374.611114:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332702720:192.168.203.156@tcp:103:rm.0 00000100:00100000:0.0:1787826374.613529:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332702976:192.168.203.156@tcp:35:rm.0 00000002:00100000:0.0:1787826374.613569:0:226914:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9057fb238380 x1874663332701568/t4294985500(4294985500) o101->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 608/696 e 0 to 0 dl 1787826390 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787826374.614017:0:226914:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.614040:0:226914:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.614101:0:226914:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801cb4 lrc: 1/0,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c8f4 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.614110:0:226914:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.614117:0:226914:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.614129:0:226914:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787826374.614133:0:226914:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057edd2c200/0x95194e8760801cb4 lrc: 0/0,0 mode: --/CR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c8f4 expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.614146:0:226914:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3e00/0x95194e8760801c98 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c8bc expref: -99 pid: 226914 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826374.614155:0:226914:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9057d128a800 lock: ffff9057d86d3800/0x95194e8760801c44 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c79d expref: -99 pid: 226875 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826374.614178:0:226914:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826374.614197:0:226914:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332703232/t0(0) o36->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826374.614256:0:226914:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332703232:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826374.614339:0:226914:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b516c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787826374.617160:0:22705:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826374.617183:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346361600 00000100:00100000:3.0:1787826374.617198:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346361600 00000100:00100000:3.0:1787826374.617284:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43704 00000100:00100000:3.0:1787826374.617297:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c5063850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346361600:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826374.617317:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826374.617410:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826374.617419:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c5063850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:125377:x1874663346361600:12345-192.168.203.156@tcp:104: Request processed in 125us (378us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826374.617428:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43704 00010000:00010000:0.0:1787826374.617439:0:6167:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826374.617450:0:6167:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617464:0:6167:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9e00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826374.617472:0:6167:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617487:0:6167:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617511:0:6167:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617583:0:6167:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617604:0:6167:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826374.617644:0:6167:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617654:0:6167:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801c60 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7dc expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826374.617682:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.617686:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.617690:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826374.617730:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332703360:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826374.620392:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332703360:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787826374.623138:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826374.623864:0:34879:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826374.623873:0:34879:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.623882:0:34879:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826374.623887:0:34879:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.623902:0:34879:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.623910:0:34879:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.624695:0:34879:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 2/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.624708:0:34879:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826374.624748:0:34879:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.624757:0:34879:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801c67 lrc: 0/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xe21309aa81f4c7f8 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826374.624777:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.624779:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826374.624781:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826374.628853:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346362752 00000100:00100000:3.0:1787826374.628866:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346362752 00000100:00100000:0.0:1787826374.641479:0:226914:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:226914:1874663332703232:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826374.641652:0:226914:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000403:0x1e5e:0x0](ffff9057dfe121c8) 00000100:00100000:3.0:1787826375.086546:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332703872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.086672:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.086702:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.086734:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1787826375.086753:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332704000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.086778:0:2395:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.092507:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2395:1874663332704000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.092543:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.092559:0:2395:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9057c7539000: committing for last_committed 4294976174 gen 19 00000100:00100000:3.0:1787826375.092587:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.094212:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2395:1874663332703872:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.094283:0:2395:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2395:1874663332704128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826375.187711:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346364416 00000100:00100000:3.0:1787826375.187725:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346364416 00000100:00100000:3.0:1787826375.187808:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43707 00000100:00100000:3.0:1787826375.187820:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:24218:x1874663346364416:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826375.187842:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826375.187933:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826375.187940:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:24218:x1874663346364416:12345-192.168.203.156@tcp:104: Request processed in 123us (438us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826375.187948:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43707 00010000:00010000:0.0:1787826375.188210:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826375.188236:0:15673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000010000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826375.188247:0:15673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057edd2ce00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826375.188255:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400010000 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826375.188284:0:15673:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e5f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787826375.188355:0:15673:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e5f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787826375.188413:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 2/0,0 mode: --/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826375.188422:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826375.188451:0:15673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826375.188460:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057c7539000 lock: 00000000d370bf05/0x95194e8760801c2f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x4d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400010020 nid: local remote: 0xe21309aa81f4c75e expref: -99 pid: 226694 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826375.188555:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826375.188558:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826375.188560:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826375.188580:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332704512:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826375.459353:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826376.482744:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826377.513372:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826380.197858:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332704640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.197988:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332704896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198032:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332705152:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198061:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705408:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198088:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705664:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198120:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:0.0:1787826380.198127:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332704768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198153:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332705024:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198177:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705280:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.198203:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705536:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205504:0:2394:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057d128a800: committing for last_committed 8589957501 gen 1 00000100:00100000:0.0:1787826380.205539:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332704768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205557:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332705024:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205567:0:2394:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9057c7539000: committing for last_committed 8589957500 gen 1 00000100:00100000:0.0:1787826380.205586:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705280:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205597:0:2394:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057c7539000: committing for last_committed 4294973865 gen 19 00000100:00100000:0.0:1787826380.205607:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705536:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205617:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332704640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205626:0:2394:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9057d128a800: committing for last_committed 4294985503 gen 1 00000100:00100000:0.0:1787826380.205639:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332704896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205663:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332705152:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205674:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705408:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826380.205686:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332705664:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826380.579504:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826382.633304:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1787826382.737810:0:227480:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:26:20 (1787826380) 00010000:00010000:3.0:1787826382.865118:0:227484:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057c7539000 00010000:00010000:3.0:1787826382.865166:0:227484:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9057d128a800 00010000:00010000:3.0:1787826382.865196:0:227484:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787826382.868686:0:227486:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff9057d128a800 00010000:00010000:0.0:1787826382.869724:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826382.869735:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 2/0,0 mode: PR/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826382.869820:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9057c7539000 lock: ffff9058021f9400/0x95194e8760801c6e lrc: 1/0,0 mode: --/PR res: [0x240000403:0x1e5e:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4c806 expref: -99 pid: 226877 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826382.869829:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1787826382.869861:0:15673:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_05:fc0b407b-a541-420d-8712-b9c8da18fe8c:15673:1874663332705920:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826382.869976:0:15673:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb376300 going to sleep for 11 seconds 00010000:00010000:0.0:1787826383.650812:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826384.674407:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826385.314459:0:160397:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:0.0:1787826385.314466:0:160397:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9057d128a800: disconnect after 20s idle 00000100:00100000:0.0:1787826385.314548:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706176:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314642:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706560:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314668:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706816:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314690:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332707072:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314717:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:0.0:1787826385.314724:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332706048:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314750:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706304:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314772:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706688:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.314795:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706944:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330482:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332706048:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330505:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706304:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330517:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706688:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330529:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706944:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330541:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706176:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330552:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332706560:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330563:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332706816:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826385.330573:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332707072:192.168.203.156@tcp:400:kworker.0 00000100:00080000:3.0:1787826385.463175:0:2396:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9057d0de2d80 x1874663332708096/t0(0) o2->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1787826385.474994:0:2392:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1787826385.475004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1787826385.475018:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1787826385.475023:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1787826385.475038:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000bc74db2c): req@ffff9057d0de2d80 x1874663332708096/t0(0) o2->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826385.475068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332707968:192.168.203.156@tcp:8:touch.0 00010000:00010000:3.0:1787826385.702103:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826386.857743:0:160397:0:(vvp_io.c:1826:vvp_io_init()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826386.858012:0:160397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826386.858048:0:160397:0:(vvp_io.c:304:vvp_io_fini()) [0x240000403:0x1e59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787826387.366635:0:227665:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826387.366662:0:227665:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826387.366911:0:227665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 227665 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826387.366923:0:227665:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 227665 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826387.366935:0:227665:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 227665 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826387.367012:0:227665:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mkdir:fc0b407b-a541-420d-8712-b9c8da18fe8c:227665:1874663332708992:192.168.203.156@tcp:101:mkdir.0 00000100:00100000:3.0:1787826387.367106:0:227665:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9b5da80 going to sleep for 11 seconds 00010000:00010000:0.0:1787826388.772321:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826389.041880:0:227665:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job mkdir:fc0b407b-a541-420d-8712-b9c8da18fe8c:227665:1874663332708992:192.168.203.156@tcp:101:mkdir.0 00010000:00010000:0.0:1787826389.041904:0:227665:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041917:0:227665:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041928:0:227665:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041938:0:227665:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041962:0:227665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041972:0:227665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.041979:0:227665:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826389.041991:0:227665:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb222300 x1874663332708992/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826403 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826389.042004:0:227665:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.042017:0:227665:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826389.042024:0:227665:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826389.042051:0:227665:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826389.042058:0:227665:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801cec 00000080:00010000:0.0:1787826389.042063:0:227665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826389.042066:0:227665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cec from it ffffb512c1d6fbe0 00010000:00010000:0.0:1787826389.042069:0:227665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.042079:0:227665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.042102:0:227665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826389.042127:0:227665:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46a.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=1536 00000080:00200000:0.0:1787826389.042132:0:227665:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826389.042145:0:227665:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.042155:0:227665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.042164:0:227665:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.042170:0:227665:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.042175:0:227665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826389.042178:0:227665:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801cec 00000080:00010000:0.0:1787826389.042181:0:227665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cec from it ffffb512c1d6fce0 00010000:00010000:0.0:1787826389.042185:0:227665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.042193:0:227665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.042201:0:227665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826389.042207:0:227665:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826389.042213:0:227665:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.042263:0:227665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.042273:0:227665:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.042279:0:227665:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.042284:0:227665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826389.042286:0:227665:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801cec 00000080:00010000:0.0:1787826389.042290:0:227665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cec from it ffffb512c1d6fdc8 00010000:00010000:0.0:1787826389.042293:0:227665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.042301:0:227665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826389.042309:0:227665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826389.042315:0:227665:0:(namei.c:2239:do_mkdir()) VFS Op:name=f46a.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988) 00800000:00000004:0.0:1787826389.042339:0:227665:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff9057c7539000: age 2499, max_age 0 00000100:00100000:0.0:1787826389.042604:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826389.042622:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332709120:192.168.203.156@tcp:41:mkdir.0 00000100:00100000:0.0:1787826389.046254:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332709120:192.168.203.156@tcp:41:mkdir.0 00000100:00100000:3.0:1787826389.046925:0:227665:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238e00 pname:cluuid:pid:xid:nid:opc:job mkdir:fc0b407b-a541-420d-8712-b9c8da18fe8c:227665:1874663332709376:192.168.203.156@tcp:36:mkdir.0 00000080:00200000:0.0:1787826389.717486:0:227347:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.717511:0:227347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.717526:0:227347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.717537:0:227347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.717544:0:227347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.717550:0:227347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.717553:0:227347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801cf3 00000080:00010000:0.0:1787826389.717559:0:227347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cf3 from it ffffb512c14b3cd8 00010000:00010000:0.0:1787826389.717562:0:227347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.717570:0:227347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826389.717773:0:227347:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988) pos/size0/73728 32bit_api 0 00000080:00200000:0.0:1787826389.717842:0:227347:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988) pos/size18446744073709551614/73728 32bit_api 0 00000080:00200008:0.0:1787826389.717859:0:227347:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), flags 304000 00000100:00100000:0.0:1787826389.722706:0:227707:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edd04b40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826389.734321:0:227707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332709760:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826389.734352:0:227707:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734375:0:227707:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734383:0:227707:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734395:0:227707:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734405:0:227707:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734420:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734428:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734435:0:227707:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826389.734448:0:227707:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb239500 x1874663332709760/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826405 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826389.734460:0:227707:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826389.734480:0:227707:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734490:0:227707:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1768 (0 0) 00000080:00200000:0.0:1787826389.734497:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826389.734544:0:227707:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6e8:0x0](ffff9057dfdd5b88): rc = 0 00000080:00200000:0.0:1787826389.734559:0:227707:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e8:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1787826389.734563:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) for lock 0x95194e8760801cfa 00000080:00010000:0.0:1787826389.734588:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) 00000080:00010000:0.0:1787826389.734593:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cfa from it ffffb512c2067b20 00010000:00010000:0.0:1787826389.734597:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734606:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826389.734640:0:227707:0:(file.c:6296:ll_getattr_dentry()) START file f46a.sanityn:[0x200000402:0x6e8:0x0](ffff9057dfdd5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826389.734648:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6e8:0x0](ffff9057dfdd5b88),name=f46a.sanityn 00000002:00010000:0.0:1787826389.734687:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6e8:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.734697:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.734708:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.734715:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.734720:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) 00000080:00010000:0.0:1787826389.734723:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) for lock 0x95194e8760801cfa 00000080:00010000:0.0:1787826389.734726:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cfa from it ffffb512c2067cb8 00010000:00010000:0.0:1787826389.734730:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.734736:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.734754:0:227707:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e8:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826389.734775:0:227707:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6e8:0x0] 00000008:00010000:0.0:1787826389.734795:0:227707:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826389.734798:0:227707:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5df30, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826389.734841:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.734851:0:227707:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826389.735048:0:227707:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826389.742551:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742590:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742605:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742623:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826389.742639:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826389.742650:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826389.742833:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742843:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742850:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.742859:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826389.742878:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332709888:192.168.203.156@tcp:101:rm.0 00000080:00200000:0.0:1787826389.744802:0:227707:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6e8:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826389.744812:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_release():245 00000020:00010000:0.0:1787826389.744816:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826389.744822:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5df30 (ffff9057d889c1b8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826389.744828:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826389.744831:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5df30 (ffff9057d889c878/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826389.744848:0:227707:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6e8:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826389.744906:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.744931:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.744960:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826389.744975:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826389.744984:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.744991:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.744994:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801cf3 00000080:00010000:0.0:1787826389.745002:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cf3 from it ffffb512c2067c10 00010000:00010000:0.0:1787826389.745007:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1787826389.745016:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826389.745030:0:227707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826389.745040:0:227707:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f46a.sanityn, flags=0 00000080:00200000:0.0:1787826389.745046:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.745052:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.745059:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826389.745074:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1787826389.745081:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.745084:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.745087:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801cf3 00000080:00010000:0.0:1787826389.745090:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801cf3 from it ffffb512c2067d68 00010000:00010000:0.0:1787826389.745093:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1787826389.745101:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1787826389.745110:0:227707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826389.745115:0:227707:0:(namei.c:2418:ll_unlink()) VFS Op:name=f46a.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988) 00010000:00010000:0.0:1787826389.745864:0:227707:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826389.745977:0:227707:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.745992:0:227707:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6e8:0x0]: losing layout lock 00010000:00010000:0.0:1787826389.746004:0:227707:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801cd7 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6e8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4c9f0 expref: -99 pid: 227600 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826389.746013:0:227707:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6e8:0x0]: losing layout lock 00010000:00010000:0.0:1787826389.746046:0:227707:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.746054:0:227707:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801cd7 lrc: 1/0,0 mode: --/CR res: [0x200000402:0x6e8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c9f0 expref: -99 pid: 227600 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826389.746061:0:227707:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1787826389.746067:0:227707:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1787826389.746070:0:227707:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c800/0x95194e8760801cfa lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ca6e expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.746083:0:227707:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b07a00/0x95194e8760801cd7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6e8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4c9f0 expref: -99 pid: 227600 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826389.746098:0:227707:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2da00/0x95194e8760801cf3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ca52 expref: -99 pid: 227347 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:0.0:1787826389.746120:0:227707:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826389.746132:0:227707:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d500 x1874663332710144/t0(0) o36->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826389.747153:0:227707:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710144:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826389.747280:0:227707:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9822d80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826389.752586:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346373632 00000100:00100000:3.0:1787826389.752603:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346373632 00000100:00100000:3.0:1787826389.752693:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43708 00000100:00100000:3.0:1787826389.752706:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:35092:x1874663346373632:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826389.752749:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9000/0x95194e8760801cec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4ca3d expref: -99 pid: 227665 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787826389.753493:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826389.753505:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:35092:x1874663346373632:12345-192.168.203.156@tcp:104: Request processed in 800us (1177us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826389.753514:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43708 00000100:00100000:0.0:1787826389.772346:0:227707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710144:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826389.772476:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.772505:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.772577:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 227707 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.772590:0:227707:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 227707 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.772600:0:227707:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 227707 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826389.772624:0:227707:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710400:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826389.772736:0:227707:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9822d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826389.792039:0:227707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710400:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826389.792081:0:227707:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792097:0:227707:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792110:0:227707:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792123:0:227707:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792139:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792148:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792165:0:227707:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826389.792179:0:227707:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d180 x1874663332710400/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826405 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826389.792194:0:227707:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792210:0:227707:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826389.792274:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826389.792313:0:227707:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826389.792321:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d08 00000080:00010000:0.0:1787826389.792328:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.792332:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d08 from it ffffb512c2067b58 00010000:00010000:0.0:1787826389.792335:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.792345:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.792386:0:227707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826389.792413:0:227707:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46a.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=8 00000080:00200000:0.0:1787826389.792419:0:227707:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46a.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=getattr 00000002:00010000:0.0:1787826389.792445:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46a.sanityn-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826389.795744:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826389.807677:0:227707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710528:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826389.807714:0:227707:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807752:0:227707:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807762:0:227707:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807776:0:227707:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807788:0:227707:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807802:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807812:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807821:0:227707:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826389.807835:0:227707:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158d180 x1874663332710528/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826405 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826389.807848:0:227707:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826389.807863:0:227707:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.807876:0:227707:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1768 (0 0) 00000080:00200000:0.0:1787826389.807884:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826389.807960:0:227707:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6e8:0x0](ffff9057dfdd5b88): rc = 0 00000080:00200000:0.0:1787826389.807977:0:227707:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826389.807985:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) for lock 0x95194e8760801d0f 00000080:00010000:0.0:1787826389.807998:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) 00000080:00010000:0.0:1787826389.808002:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d0f from it ffffb512c2067b20 00010000:00010000:0.0:1787826389.808006:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826389.808017:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826389.808062:0:227707:0:(file.c:6296:ll_getattr_dentry()) START file f46a.sanityn-2:[0x200000402:0x6e8:0x0](ffff9057dfdd5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826389.808070:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6e8:0x0](ffff9057dfdd5b88),name=f46a.sanityn-2 00000002:00010000:0.0:1787826389.808094:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6e8:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.808104:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808116:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.808124:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.808130:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) 00000080:00010000:0.0:1787826389.808133:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e8:0x0](ffff9057dfdd5b88) for lock 0x95194e8760801d0f 00000080:00010000:0.0:1787826389.808137:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d0f from it ffffb512c2067cb8 00010000:00010000:0.0:1787826389.808141:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808148:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.808171:0:227707:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e8:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826389.808182:0:227707:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6e8:0x0] 00000008:00010000:0.0:1787826389.808203:0:227707:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826389.808207:0:227707:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d7b0, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826389.808597:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808614:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826389.808629:0:227707:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826389.808634:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808644:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826389.808656:0:227707:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826389.808668:0:227707:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6e8:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826389.808672:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_release():245 00000020:00010000:0.0:1787826389.808676:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826389.808679:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d7b0 (ffff9057d889c1b8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826389.808684:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89bc0 (ffff9057d889c878/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826389.808687:0:227707:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d7b0 (ffff9057d889c878/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826389.808699:0:227707:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6e8:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826389.808755:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.808766:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.808775:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808784:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826389.808791:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.808796:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.808799:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d08 00000080:00010000:0.0:1787826389.808803:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d08 from it ffffb512c2067c10 00010000:00010000:0.0:1787826389.808806:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826389.808814:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.808824:0:227707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826389.808833:0:227707:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f46a.sanityn-2, flags=0 00000080:00200000:0.0:1787826389.808838:0:227707:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826389.808845:0:227707:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826389.808851:0:227707:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.808860:0:227707:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.808867:0:227707:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826389.808871:0:227707:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826389.808873:0:227707:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d08 00000080:00010000:0.0:1787826389.808877:0:227707:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d08 from it ffffb512c2067d68 00010000:00010000:0.0:1787826389.808880:0:227707:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.808888:0:227707:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826389.808895:0:227707:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826389.808901:0:227707:0:(namei.c:2418:ll_unlink()) VFS Op:name=f46a.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988) 00010000:00010000:0.0:1787826389.808928:0:227707:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826389.808971:0:227707:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826389.808983:0:227707:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6e8:0x0]: losing layout lock 00010000:00010000:0.0:1787826389.809021:0:227707:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.809036:0:227707:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826389.809043:0:227707:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787826389.809046:0:227707:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801d0f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6e8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cab4 expref: -99 pid: 227707 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826389.809070:0:227707:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d08 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4ca98 expref: -99 pid: 227707 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787826389.809093:0:227707:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826389.809104:0:227707:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158d180 x1874663332710784/t0(0) o36->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826389.809132:0:227707:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710784:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826389.810018:0:227707:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057f9822d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787826389.820894:0:227707:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:227707:1874663332710784:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826389.820984:0:227707:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826389.821001:0:227707:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826389.821036:0:227707:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6e8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826389.821045:0:227707:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x6e8:0x0](ffff9057dfdd5b88) 00000008:00010000:0.0:1787826389.821089:0:227707:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826385 1787826385 1787826385 by oinfo size 0 blocks 0 [cma]time 1787826385 1787826385 1787826385 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787826390.437733:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332710912:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826390.452255:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332711424:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826390.452277:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332711936:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826390.455745:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332711168:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826390.455764:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332711680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826390.520481:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346373888 00000100:00100000:3.0:1787826390.520495:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346373888 00000100:00100000:3.0:1787826390.520568:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43709 00000100:00100000:3.0:1787826390.520579:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35096:x1874663346373888:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826390.520601:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000001ec42649/0x95194e8760801d01 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x524:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xe21309aa81f4ca75 expref: -99 pid: 227707 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826390.520707:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826390.520715:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:35096:x1874663346373888:12345-192.168.203.156@tcp:104: Request processed in 137us (360us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826390.520723:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43709 00010000:00010000:3.0:1787826390.818724:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826391.843339:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826392.866463:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826394.918629:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826395.562078:0:2392:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9057d0de1500 x1874663332712704/t0(0) o9->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 0 dl 1787826405 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826395.562092:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1787826395.562097:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332712704:192.168.203.156@tcp:9:kworker.0 00000100:00080000:0.0:1787826395.562111:0:2392:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9057d0de0e00 x1874663332713088/t0(0) o9->lustre-OST0000-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 0 dl 1787826405 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826395.562118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83df800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1787826395.562121:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332713088:192.168.203.156@tcp:9:kworker.0 00000100:00080000:0.0:1787826395.562133:0:2392:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9057d0de3800 x1874663332713216/t0(0) o9->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 0 dl 1787826405 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826395.562139:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1787826395.562143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332713216:192.168.203.156@tcp:9:kworker.0 00010000:00010000:0.0:1787826396.964235:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826397.095373:0:160397:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826397.095393:0:160397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826397.095421:0:160397:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000001:02000400:0.0:1787826397.685366:0:228272:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test 46b: pdirops: link vs create ======================================================================== 06:26:36 (1787826396) 00010000:00010000:0.0:1787826399.010785:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826400.889975:0:228391:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826400.890019:0:228391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826400.890102:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.890115:0:228391:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.890127:0:228391:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826400.892087:0:228391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714112:192.168.203.156@tcp:101:touch.0 00000100:00100000:0.0:1787826400.892541:0:228391:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c3d72f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826400.903083:0:228391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714112:192.168.203.156@tcp:101:touch.0 00010000:00010000:0.0:1787826400.903123:0:228391:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903150:0:228391:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903165:0:228391:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903180:0:228391:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903198:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903207:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903215:0:228391:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826400.903474:0:228391:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f480 x1874663332714112/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826416 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826400.903492:0:228391:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903508:0:228391:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826400.903516:0:228391:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826400.903552:0:228391:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826400.903559:0:228391:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d16 00000080:00010000:0.0:1787826400.903565:0:228391:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826400.903569:0:228391:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d16 from it ffffb512c1cbbbf8 00010000:00010000:0.0:1787826400.903572:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903582:0:228391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826400.903623:0:228391:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826400.903653:0:228391:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826400.903670:0:228391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826400.903679:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.903690:0:228391:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826400.903697:0:228391:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826400.903701:0:228391:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826400.903704:0:228391:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d16 00000080:00010000:0.0:1787826400.903708:0:228391:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d16 from it ffffb512c1cbbb38 00010000:00010000:0.0:1787826400.903711:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.903719:0:228391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826400.903727:0:228391:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826400.903732:0:228391:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), file ffff9057c8adfe00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1787826400.903742:0:228391:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=open|creat 00000002:00010000:0.0:1787826400.903766:0:228391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46b.sanityn-2,[0x200000402:0x6e9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1787826400.903777:0:228391:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826400.903836:0:228391:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826400.903844:0:228391:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826400.903847:0:228391:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d16 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cae5 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826400.903899:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.903907:0:228391:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826400.903920:0:228391:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826400.903928:0:228391:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158f480 x1874663332714368/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826400.903938:0:228391:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826400.903961:0:228391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714368:192.168.203.156@tcp:101:touch.0 00000100:00100000:0.0:1787826400.904493:0:228391:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c3d72f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826400.912865:0:228391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714368:192.168.203.156@tcp:101:touch.0 00010000:00010000:0.0:1787826400.912912:0:228391:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.912951:0:228391:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.912967:0:228391:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.912979:0:228391:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.912991:0:228391:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.913007:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.913016:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.913024:0:228391:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826400.913037:0:228391:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9057f158f480 x1874663332714368/t8589957518(8589957518) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826416 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787826400.913052:0:228391:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f480 x1874663332714368/t8589957518(8589957518) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 656/664 e 0 to 0 dl 1787826416 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826400.913068:0:228391:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826400.913094:0:228391:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.913107:0:228391:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1769 (0 0) 00000080:00200000:0.0:1787826400.913115:0:228391:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826400.913132:0:228391:0:(namei.c:1628:ll_create_it()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=open|creat 00000080:00200000:0.0:1787826400.913181:0:228391:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00200000:0.0:1787826400.913376:0:228391:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826400.913382:0:228391:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826400.913386:0:228391:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d1d 00000080:00010000:0.0:1787826400.913398:0:228391:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00200008:0.0:1787826400.913416:0:228391:0:(file.c:1090:ll_file_open()) START file f46b.sanityn-2:[0x200000402:0x6e9:0x0](ffff9057dfe121c8), flags 104501 00000002:00100000:0.0:1787826400.913443:0:228391:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9057f158f480 x1874663332714368/t8589957518(8589957518) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826416 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787826400.913492:0:228391:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d1d from it ffff9057f9ae14e0 00010000:00010000:0.0:1787826400.913497:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.913507:0:228391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826400.913531:0:228391:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6e9:0x0](ffff9057dfe121c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1787826400.913576:0:228391:0:(llite_lib.c:2427:ll_setattr_raw()) START file f46b.sanityn-2:[0x200000402:0x6e9:0x0](ffff9057dfe121c8) current size 0, valid attrs 20070, mode 9057, uid -1043612080, gid -19182, new size 18446744072537700887, atime 1787826400.000000000, mtime 1787826400.000000000, ctime 1787826400.000000000 00010000:00010000:0.0:1787826400.913616:0:228391:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826400.913636:0:228391:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6e9:0x0]: losing layout lock 00010000:00010000:0.0:1787826400.913682:0:228391:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.913690:0:228391:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826400.913693:0:228391:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d1d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cb01 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1787826400.913716:0:228391:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826400.913724:0:228391:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c380 x1874663332714624/t0(0) o36->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826400.913751:0:228391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714624:192.168.203.156@tcp:36:touch.0 00000100:00100000:0.0:1787826400.913846:0:228391:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c3d72180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826400.923801:0:228391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714624:192.168.203.156@tcp:36:touch.0 00000080:00200000:0.0:1787826400.923880:0:228391:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1787826400.923913:0:228391:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000002:00010000:0.0:1787826400.923926:0:228391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6e9:0x0]) in obj [0x200000402:0x6e9:0x0], intent: layout flags 0 00010000:00010000:0.0:1787826400.923990:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.924002:0:228391:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.924014:0:228391:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228391 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826400.924038:0:228391:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714752:192.168.203.156@tcp:101:touch.0 00000100:00100000:0.0:1787826400.924330:0:228391:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c3d72180 going to sleep for 11 seconds 00000100:00100000:0.0:1787826400.932825:0:228391:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:228391:1874663332714752:192.168.203.156@tcp:101:touch.0 00010000:00010000:0.0:1787826400.932866:0:228391:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.932896:0:228391:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826400.932915:0:228391:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9057f158e680 x1874663332714752/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 376/400 e 0 to 0 dl 1787826416 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826400.932927:0:228391:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826400.932942:0:228391:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.932958:0:228391:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 16 res 8589935618/1769 (0 0) 00000080:00200000:0.0:1787826400.932967:0:228391:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000402:0x6e9:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1787826400.932991:0:228391:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d24 00000080:00010000:0.0:1787826400.933000:0:228391:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d24 from it ffffb512c1cbba78 00010000:00010000:0.0:1787826400.933006:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826400.933018:0:228391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.933049:0:228391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.933068:0:228391:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1787826400.933092:0:228391:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000402:0x6e9:0x0](00000000611956bd) being reconfigured ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826400.933141:0:228391:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e9:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1787826400.933147:0:228391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826400.933155:0:228391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d24 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6e9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cb16 expref: -99 pid: 228391 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1787826400.933178:0:228391:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00080000:0.0:1787826400.933279:0:228391:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8134 00000100:00080000:0.0:1787826400.933294:0:228391:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826400.933306:0:228391:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8437760, lost: 0. 00000100:00100000:0.0:1787826400.933334:0:228391:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332714880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826400.947359:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:0:1874663332715008:192.168.203.156@tcp:2:touch.0 00010000:00010000:0.0:1787826401.059270:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826402.082454:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826403.034060:0:228456:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826403.034105:0:228456:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826403.034206:0:228456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826403.034274:0:228456:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826403.035737:0:228456:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826403.035784:0:228456:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job multiop:fc0b407b-a541-420d-8712-b9c8da18fe8c:228456:1874663332715904:192.168.203.156@tcp:101:multiop.0 00000100:00100000:0.0:1787826403.036022:0:228456:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c3d72180 going to sleep for 11 seconds 00010000:00010000:0.0:1787826403.106522:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826405.011821:0:228456:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job multiop:fc0b407b-a541-420d-8712-b9c8da18fe8c:228456:1874663332715904:192.168.203.156@tcp:101:multiop.0 00010000:00010000:0.0:1787826405.011855:0:228456:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011874:0:228456:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011885:0:228456:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011895:0:228456:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011907:0:228456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011914:0:228456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011926:0:228456:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826405.011936:0:228456:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332715904/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826419 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826405.011949:0:228456:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.011959:0:228456:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826405.011965:0:228456:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826405.011992:0:228456:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826405.011998:0:228456:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801d39 00000080:00010000:0.0:1787826405.012003:0:228456:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826405.012006:0:228456:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d39 from it ffffb512c2007bf8 00010000:00010000:0.0:1787826405.012010:0:228456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.012019:0:228456:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.012044:0:228456:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826405.012070:0:228456:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826405.012085:0:228456:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.012093:0:228456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.012108:0:228456:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.012114:0:228456:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826405.012119:0:228456:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826405.012121:0:228456:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801d39 00000080:00010000:0.0:1787826405.012125:0:228456:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d39 from it ffffb512c2007b38 00010000:00010000:0.0:1787826405.012128:0:228456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.012134:0:228456:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.012142:0:228456:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826405.012147:0:228456:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f46b.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), file ffff9057c6273380, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787826405.012166:0:228456:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46b.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=open|creat 00000002:00010000:0.0:1787826405.012189:0:228456:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46b.sanityn,[0x200000403:0x1dc4:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787826405.012200:0:228456:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.012266:0:228456:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.012273:0:228456:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826405.012276:0:228456:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2000/0x95194e8760801d39 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cb63 expref: -99 pid: 228456 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.012315:0:228456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.012323:0:228456:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787826405.012333:0:228456:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787826405.012340:0:228456:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332716160/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787826405.012350:0:228456:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826405.012367:0:228456:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job multiop:fc0b407b-a541-420d-8712-b9c8da18fe8c:228456:1874663332716160:192.168.203.156@tcp:101:multiop.0 00000100:00100000:0.0:1787826405.012509:0:228456:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2ab480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826405.033199:0:228456:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job multiop:fc0b407b-a541-420d-8712-b9c8da18fe8c:228456:1874663332716160:192.168.203.156@tcp:101:multiop.0 00010000:00010000:0.0:1787826405.033266:0:228456:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033282:0:228456:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033289:0:228456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033297:0:228456:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033306:0:228456:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033314:0:228456:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3c00 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826405.033318:0:228456:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033326:0:228456:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x206400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033333:0:228456:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x20e400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033341:0:228456:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787826405.033347:0:228456:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.033354:0:228456:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3c00/0x95194e8760801d40 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 228456 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826405.033392:0:228456:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9057f158ed80 x1874663332716160/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 656/608 e 0 to 0 dl 1787826421 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787826405.033407:0:228456:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:0.0:1787826405.154473:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826405.579199:0:228140:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:228140:1874663332716416:192.168.203.156@tcp:37:bash.0 00000100:00100000:3.0:1787826405.579384:0:228140:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057fb3f4a80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826405.591882:0:228140:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:228140:1874663332716416:192.168.203.156@tcp:37:bash.0 00010000:00010000:3.0:1787826405.594121:0:228140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e8760801d47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cb94 expref: -99 pid: 228140 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826405.594149:0:228140:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057edd2c400/0x95194e8760801d47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cb94 expref: -99 pid: 228140 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826405.595444:0:228140:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000100:00100000:0.0:1787826405.618084:0:228498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332716544:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826405.618137:0:228498:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618159:0:228498:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618173:0:228498:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618186:0:228498:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618199:0:228498:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618215:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618268:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618277:0:228498:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826405.618290:0:228498:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0a80 x1874663332716544/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826421 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826405.618302:0:228498:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826405.618317:0:228498:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618334:0:228498:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1769 (0 0) 00000080:00200000:0.0:1787826405.618342:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826405.618409:0:228498:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6e9:0x0](ffff9057dfe121c8): rc = 0 00000080:00200000:0.0:1787826405.618425:0:228498:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e9:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1787826405.618431:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d4e 00000080:00010000:0.0:1787826405.618444:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826405.618448:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d4e from it ffffb512c214fb20 00010000:00010000:0.0:1787826405.618452:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618462:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826405.618522:0:228498:0:(file.c:6296:ll_getattr_dentry()) START file f46b.sanityn:[0x200000402:0x6e9:0x0](ffff9057dfe121c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826405.618529:0:228498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6e9:0x0](ffff9057dfe121c8),name=f46b.sanityn 00000002:00010000:0.0:1787826405.618557:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6e9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.618567:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.618579:0:228498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.618585:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826405.618591:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826405.618595:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d4e 00000080:00010000:0.0:1787826405.618599:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d4e from it ffffb512c214fcb8 00010000:00010000:0.0:1787826405.618603:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.618611:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06000/0x95194e8760801d4e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbb0 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.618633:0:228498:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e9:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826405.618644:0:228498:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6e9:0x0] 00000008:00010000:0.0:1787826405.618666:0:228498:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826405.618670:0:228498:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d730, osc lock ffff9057c8b82e40, flags 0x400001002 00010000:00010000:0.0:1787826405.618727:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.618738:0:228498:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826405.618776:0:228498:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826405.618791:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826405.618812:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332716672:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826405.626855:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.626890:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.626907:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.626927:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826405.626945:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826405.626957:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826405.626977:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.626986:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.626994:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.627005:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826405.627029:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332716672:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826405.636452:0:228498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332716928:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826405.636699:0:228498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826405.636730:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.636803:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.636816:0:228498:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.636828:0:228498:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826405.636862:0:228498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717056:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826405.636976:0:228498:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2ab480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826405.649781:0:228498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717056:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826405.649819:0:228498:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649845:0:228498:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649861:0:228498:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649875:0:228498:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649891:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649900:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.649913:0:228498:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826405.649926:0:228498:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332717056/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826421 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826405.649941:0:228498:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.650023:0:228498:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826405.650034:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826405.650083:0:228498:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826405.650293:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d5c 00000080:00010000:0.0:1787826405.650303:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826405.650306:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d5c from it ffffb512c214fb58 00010000:00010000:0.0:1787826405.650310:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.650321:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.650352:0:228498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826405.650457:0:228498:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=8 00000080:00200000:0.0:1787826405.650463:0:228498:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=getattr 00000002:00010000:0.0:1787826405.650587:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46b.sanityn-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826405.650642:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.650652:0:228498:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.650663:0:228498:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 228498 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826405.650691:0:228498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717184:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826405.650894:0:228498:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2ab480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826405.656078:0:228498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717184:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826405.656127:0:228498:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656159:0:228498:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656168:0:228498:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656181:0:228498:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656193:0:228498:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656208:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656266:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656275:0:228498:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826405.656287:0:228498:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332717184/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826421 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826405.656299:0:228498:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826405.656311:0:228498:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656323:0:228498:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1769 (0 0) 00000080:00200000:0.0:1787826405.656330:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826405.656393:0:228498:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6e9:0x0](ffff9057dfe121c8): rc = 0 00000080:00200000:0.0:1787826405.656407:0:228498:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6e9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826405.656413:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d63 00000080:00010000:0.0:1787826405.656427:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826405.656430:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d63 from it ffffb512c214fb20 00010000:00010000:0.0:1787826405.656434:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826405.656444:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826405.656625:0:228498:0:(file.c:6296:ll_getattr_dentry()) START file f46b.sanityn-2:[0x200000402:0x6e9:0x0](ffff9057dfe121c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826405.656634:0:228498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6e9:0x0](ffff9057dfe121c8),name=f46b.sanityn-2 00000002:00010000:0.0:1787826405.656660:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6e9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.656670:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.656682:0:228498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.656689:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826405.656695:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826405.656698:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e9:0x0](ffff9057dfe121c8) for lock 0x95194e8760801d63 00000080:00010000:0.0:1787826405.656703:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d63 from it ffffb512c214fcb8 00010000:00010000:0.0:1787826405.656706:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.656714:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.656736:0:228498:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e9:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826405.656747:0:228498:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6e9:0x0] 00000008:00010000:0.0:1787826405.656768:0:228498:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826405.656772:0:228498:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5dd30, osc lock ffff9057c8b82be0, flags 0x400001002 00010000:00010000:0.0:1787826405.656793:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.656812:0:228498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787826405.656824:0:228498:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787826405.656829:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.656847:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787826405.656859:0:228498:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787826405.656870:0:228498:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6e9:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826405.656874:0:228498:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_release():245 00000020:00010000:0.0:1787826405.656878:0:228498:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826405.656881:0:228498:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5dd30 (ffff9057f989ccf8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826405.656886:0:228498:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f8b780 (ffff9057d889cc68/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826405.656889:0:228498:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5dd30 (ffff9057d889cc68/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826405.656901:0:228498:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6e9:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826405.656949:0:228498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826405.657196:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.657205:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.657215:0:228498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826405.657269:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826405.657274:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826405.657277:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d5c 00000080:00010000:0.0:1787826405.657282:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d5c from it ffffb512c214fc10 00010000:00010000:0.0:1787826405.657286:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826405.657294:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.657304:0:228498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826405.657314:0:228498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f46b.sanityn-2, flags=0 00000080:00200000:0.0:1787826405.657320:0:228498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826405.657326:0:228498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826405.657332:0:228498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.657340:0:228498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.657346:0:228498:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826405.657349:0:228498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826405.657351:0:228498:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d5c 00000080:00010000:0.0:1787826405.657480:0:228498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d5c from it ffffb512c214fd68 00010000:00010000:0.0:1787826405.657547:0:228498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.657562:0:228498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826405.657571:0:228498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826405.657578:0:228498:0:(namei.c:2418:ll_unlink()) VFS Op:name=f46b.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988) 00010000:00010000:0.0:1787826405.657604:0:228498:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826405.657632:0:228498:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826405.657642:0:228498:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6e9:0x0]: losing layout lock 00010000:00010000:0.0:1787826405.657679:0:228498:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.657686:0:228498:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826405.657693:0:228498:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787826405.657696:0:228498:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2e00/0x95194e8760801d63 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4cbf6 expref: -99 pid: 228498 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826405.657718:0:228498:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2000/0x95194e8760801d5c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cbda expref: -99 pid: 228498 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787826405.657739:0:228498:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826405.657751:0:228498:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158ed80 x1874663332717440/t0(0) o36->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826405.657779:0:228498:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717440:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826405.657912:0:228498:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2ab480 going to sleep for 11 seconds 00000100:00100000:0.0:1787826405.663062:0:228498:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:228498:1874663332717440:192.168.203.156@tcp:36:rm.0 00000080:00200000:0.0:1787826405.663172:0:228498:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6e9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1787826405.663188:0:228498:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787826405.664326:0:228498:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6e9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826405.664344:0:228498:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x6e9:0x0](ffff9057dfe121c8) 00000008:00010000:0.0:1787826405.664404:0:228498:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1787826400 1787826400 1787826400 by oinfo size 0 blocks 0 [cma]time 1787826400 1787826400 1787826400 ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 1/0,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787826405.811975:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332717696:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826405.812002:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332717952:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826405.812018:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332717824:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826405.812031:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332718080:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826407.204181:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826407.331338:0:160397:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787826407.331359:0:160397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787826407.331394:0:160397:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1dc3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1787826409.250557:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826410.689402:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00010000:00010000:0.0:1787826410.689875:0:15673:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826410.689885:0:15673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826410.689896:0:15673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d3c00 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826410.689902:0:15673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826410.689959:0:15673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826410.689969:0:15673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826410.690005:0:15673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826410.690015:0:15673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057d128a800 lock: 0000000066d21abb/0x95194e8760801d55 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4cbb7 expref: -99 pid: 228498 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826410.690044:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826410.690047:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826410.690049:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826410.690059:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787826410.690078:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332718208:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826410.700095:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332718208:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00080000:3.0:1787826410.927472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1787826410.927482:0:149405:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9057d128a800: disconnect after 20s idle 00000100:00100000:3.0:1787826410.927550:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332718464:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.927665:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332718848:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.927699:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1787826410.927705:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332718336:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.927729:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332718592:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.927749:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332718976:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.949789:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332718848:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826410.953055:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332718592:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826411.298411:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826412.325027:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826414.370420:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826415.752833:0:229181:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826415.752878:0:229181:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000100:00100000:0.0:1787826415.792317:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332719872:192.168.203.156@tcp:8:touch.0 00000100:00100000:0.0:1787826415.798098:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9057d128a800: committing for last_committed 4294973867 gen 33 00000100:00080000:0.0:1787826415.798124:0:2392:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff9057d128a800: connect to target with instance 1 00000100:00080000:0.0:1787826415.798138:0:2392:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff9057d128a800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1787826415.798157:0:2392:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1787826415.798161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1787826415.798175:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1787826415.798181:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1787826415.798202:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000003169dadf): req@ffff9057fb239f80 x1874663332720000/t0(0) o2->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826415.805577:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332719872:192.168.203.156@tcp:8:touch.0 00000100:00100000:0.0:1787826416.047598:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332720768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826416.056024:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332720384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826416.056051:0:2393:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9057d128a800: committing for last_committed 4294976176 gen 39 00000100:00100000:0.0:1787826416.056072:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332720640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826416.056088:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332720896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826416.057129:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332720512:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826416.057145:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332720768:192.168.203.156@tcp:400:kworker.0 00000080:00200000:3.0:1787826417.210398:0:229244:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826417.210429:0:229244:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826417.212400:0:229244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826417.212415:0:229244:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826417.212432:0:229244:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826417.212459:0:229244:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:229244:1874663332721024:192.168.203.156@tcp:101:link.0 00000100:00100000:3.0:1787826417.212620:0:229244:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edd043c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826417.225988:0:229244:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:229244:1874663332721024:192.168.203.156@tcp:101:link.0 00010000:00010000:0.0:1787826417.226015:0:229244:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226027:0:229244:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226037:0:229244:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226049:0:229244:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226062:0:229244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226071:0:229244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226078:0:229244:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826417.226090:0:229244:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220e00 x1874663332721024/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826433 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826417.226105:0:229244:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226117:0:229244:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826417.226124:0:229244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826417.226155:0:229244:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826417.226161:0:229244:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d7f 00000080:00010000:0.0:1787826417.226167:0:229244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826417.226170:0:229244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d7f from it ffffb512c20c7bd8 00010000:00010000:0.0:1787826417.226174:0:229244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226189:0:229244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826417.226213:0:229244:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826417.226278:0:229244:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=8 00000080:00200000:0.0:1787826417.226283:0:229244:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46c.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=getattr 00000002:00010000:0.0:1787826417.226304:0:229244:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46c.sanityn-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826417.226351:0:229244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226358:0:229244:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.226366:0:229244:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229244 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826417.226386:0:229244:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:229244:1874663332721152:192.168.203.156@tcp:101:link.0 00000100:00100000:0.0:1787826417.226527:0:229244:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057c6c25300 going to sleep for 11 seconds 00000100:00100000:0.0:1787826417.236873:0:229244:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:229244:1874663332721152:192.168.203.156@tcp:101:link.0 00010000:00010000:0.0:1787826417.236899:0:229244:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236914:0:229244:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236921:0:229244:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236933:0:229244:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236948:0:229244:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236962:0:229244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236971:0:229244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.236979:0:229244:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826417.236990:0:229244:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f800 x1874663332721152/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826433 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826417.237002:0:229244:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826417.237018:0:229244:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.237037:0:229244:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1770 (0 0) 00000080:00200000:0.0:1787826417.237044:0:229244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826417.237090:0:229244:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6ea:0x0](ffff9057dfe1d348): rc = 0 00000080:00200000:0.0:1787826417.237104:0:229244:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6ea:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1787826417.237109:0:229244:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) for lock 0x95194e8760801d86 00000080:00010000:0.0:1787826417.237120:0:229244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) 00000080:00010000:0.0:1787826417.237123:0:229244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d86 from it ffffb512c20c7ba0 00010000:00010000:0.0:1787826417.237127:0:229244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826417.237135:0:229244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3600/0x95194e8760801d86 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cc89 expref: -99 pid: 229244 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826417.237183:0:229244:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826417.237198:0:229244:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826417.237208:0:229244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826417.237218:0:229244:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826417.237386:0:229244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826417.237391:0:229244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826417.237394:0:229244:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d7f 00000080:00010000:0.0:1787826417.237398:0:229244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d7f from it ffffb512c20c7bc0 00010000:00010000:0.0:1787826417.237401:0:229244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826417.237408:0:229244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9200/0x95194e8760801d7f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cc6d expref: -99 pid: 229244 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826417.237421:0:229244:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826417.717041:0:229246:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826417.717076:0:229246:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826417.717139:0:229246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826417.717150:0:229246:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826417.717160:0:229246:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826417.717186:0:229246:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job link:fc0b407b-a541-420d-8712-b9c8da18fe8c:229246:1874663332721536:192.168.203.156@tcp:101:link.0 00000100:00100000:3.0:1787826417.717304:0:229246:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edd04a80 going to sleep for 11 seconds 00010000:00010000:3.0:1787826418.469314:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826419.437832:0:229246:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job link:fc0b407b-a541-420d-8712-b9c8da18fe8c:229246:1874663332721536:192.168.203.156@tcp:101:link.0 00010000:00010000:3.0:1787826419.437869:0:229246:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437884:0:229246:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437898:0:229246:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437911:0:229246:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437925:0:229246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437933:0:229246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437940:0:229246:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826419.437965:0:229246:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221f80 x1874663332721536/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826433 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826419.437982:0:229246:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.437997:0:229246:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826419.438006:0:229246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826419.438044:0:229246:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826419.438051:0:229246:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801d8d 00000080:00010000:3.0:1787826419.438058:0:229246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826419.438062:0:229246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d8d from it ffffb512c1523bd8 00010000:00010000:3.0:1787826419.438066:0:229246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.438445:0:229246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826419.438483:0:229246:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826419.438510:0:229246:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46c.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=8 00000080:00200000:3.0:1787826419.438515:0:229246:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46c.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:3.0:1787826419.438545:0:229246:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46c.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1787826419.438599:0:229246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.438609:0:229246:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.438620:0:229246:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229246 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826419.438646:0:229246:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job link:fc0b407b-a541-420d-8712-b9c8da18fe8c:229246:1874663332721664:192.168.203.156@tcp:101:link.0 00000100:00100000:3.0:1787826419.438777:0:229246:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057edd04a80 going to sleep for 11 seconds 00000100:00100000:3.0:1787826419.452870:0:229246:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job link:fc0b407b-a541-420d-8712-b9c8da18fe8c:229246:1874663332721664:192.168.203.156@tcp:101:link.0 00010000:00010000:3.0:1787826419.452910:0:229246:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.452938:0:229246:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.452948:0:229246:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453456:0:229246:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453517:0:229246:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453536:0:229246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453546:0:229246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453554:0:229246:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826419.453637:0:229246:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb221f80 x1874663332721664/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826435 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826419.453653:0:229246:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1787826419.453670:0:229246:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453684:0:229246:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935618/1770 (0 0) 00000080:00200000:3.0:1787826419.453693:0:229246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787826419.453753:0:229246:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6ea:0x0](ffff9057dfe21988) 00000080:00200000:3.0:1787826419.453856:0:229246:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6ea:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787826419.453863:0:229246:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe21988) for lock 0x95194e8760801d94 00000080:00010000:3.0:1787826419.453877:0:229246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe21988) 00000080:00010000:3.0:1787826419.453882:0:229246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d94 from it ffffb512c1523ba0 00010000:00010000:3.0:1787826419.453885:0:229246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826419.453895:0:229246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826419.454148:0:229246:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:3.0:1787826419.454172:0:229246:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826419.454184:0:229246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.454418:0:229246:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826419.454426:0:229246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787826419.454433:0:229246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:3.0:1787826419.454436:0:229246:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801d8d 00000080:00010000:3.0:1787826419.454441:0:229246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d8d from it ffffb512c1523bc0 00010000:00010000:3.0:1787826419.454444:0:229246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826419.454452:0:229246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787826419.454540:0:229246:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:3.0:1787826419.454554:0:229246:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f46c.sanityn, flags=1536 00000080:00200000:0.0:1787826420.052749:0:228930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826420.052796:0:228930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826420.053570:0:228930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 228930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.053587:0:228930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.053600:0:228930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 228930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826420.054143:0:228930:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:228930:1874663332721792:192.168.203.156@tcp:101:bash.0 00000100:00100000:0.0:1787826420.054440:0:228930:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057eddba6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826420.067342:0:228930:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:228930:1874663332721792:192.168.203.156@tcp:101:bash.0 00010000:00010000:0.0:1787826420.067384:0:228930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067399:0:228930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067414:0:228930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067427:0:228930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067447:0:228930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067457:0:228930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067465:0:228930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826420.067479:0:228930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158f800 x1874663332721792/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826420.067494:0:228930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067509:0:228930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826420.067519:0:228930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826420.067556:0:228930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826420.067564:0:228930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d9b 00000080:00010000:0.0:1787826420.067571:0:228930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826420.067575:0:228930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d9b from it ffffb512c0dd3bb0 00010000:00010000:0.0:1787826420.067578:0:228930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.067588:0:228930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826420.067626:0:228930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 44 00000080:00200008:0.0:1787826420.067643:0:228930:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), flags 304000 00000080:00200008:0.0:1787826420.067678:0:228930:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 2047, flags 0, foreign 0 00000002:00010000:0.0:1787826420.067708:0:228930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: readdir flags 0 00010000:00010000:0.0:1787826420.067719:0:228930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.067731:0:228930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826420.068486:0:228930:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:228930:1874663332721920:192.168.203.156@tcp:37:bash.0 00000100:00100000:0.0:1787826420.071480:0:228930:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057eddba6c0 going to sleep for 11 seconds 00000080:00200000:0.0:1787826420.114052:0:229287:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826420.114093:0:229287:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826420.114117:0:229287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826420.114136:0:229287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826420.114145:0:229287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826420.114155:0:229287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826420.114160:0:229287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801d9b 00000080:00010000:0.0:1787826420.114169:0:229287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801d9b from it ffffb512c20ffb58 00010000:00010000:0.0:1787826420.114174:0:229287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826420.114191:0:229287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d2400/0x95194e8760801d9b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccd6 expref: -99 pid: 228930 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826420.114211:0:229287:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826420.114500:0:229287:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46c.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=8 00000080:00200000:0.0:1787826420.114507:0:229287:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46c.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=getattr 00000002:00010000:0.0:1787826420.114525:0:229287:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46c.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826420.114588:0:229287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 229287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.114597:0:229287:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229287 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.114607:0:229287:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 229287 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826420.114709:0:229287:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:229287:1874663332722048:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826420.114949:0:229287:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057eddba6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787826420.121470:0:229287:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job rm:398e0287-d3c7-44dd-b1d9-6b85f698e615:229287:1874663332722048:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826420.121507:0:229287:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121530:0:229287:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121538:0:229287:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121553:0:229287:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121565:0:229287:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121580:0:229287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121589:0:229287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121597:0:229287:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826420.121610:0:229287:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158dc00 x1874663332722048/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826436 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826420.121623:0:229287:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826420.121714:0:229287:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121730:0:229287:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1770 (0 0) 00000080:00200000:0.0:1787826420.121748:0:229287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826420.121815:0:229287:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6ea:0x0](ffff9057dfe1d348): rc = 0 00000080:00200000:0.0:1787826420.121831:0:229287:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6ea:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1787826420.121837:0:229287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) for lock 0x95194e8760801da2 00000080:00010000:0.0:1787826420.121852:0:229287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) 00000080:00010000:0.0:1787826420.121857:0:229287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801da2 from it ffffb512c20ffb20 00010000:00010000:0.0:1787826420.121860:0:229287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.121871:0:229287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826420.121919:0:229287:0:(file.c:6296:ll_getattr_dentry()) START file f46c.sanityn:[0x200000402:0x6ea:0x0](ffff9057dfe1d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826420.121927:0:229287:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6ea:0x0](ffff9057dfe1d348),name=f46c.sanityn 00000002:00010000:0.0:1787826420.121954:0:229287:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6ea:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826420.121965:0:229287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.121978:0:229287:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826420.121985:0:229287:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826420.121990:0:229287:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) 00000080:00010000:0.0:1787826420.121994:0:229287:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6ea:0x0](ffff9057dfe1d348) for lock 0x95194e8760801da2 00000080:00010000:0.0:1787826420.121998:0:229287:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801da2 from it ffffb512c20ffcb8 00010000:00010000:0.0:1787826420.122001:0:229287:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.122010:0:229287:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3c00/0x95194e8760801da2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ccf2 expref: -99 pid: 229287 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826420.122037:0:229287:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6ea:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826420.122048:0:229287:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6ea:0x0] 00000008:00010000:0.0:1787826420.122070:0:229287:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826420.122074:0:229287:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d7b0, osc lock ffff9057c8b82be0, flags 0x400001002 00010000:00010000:0.0:1787826420.123769:0:229287:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.123791:0:229287:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826420.123839:0:229287:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1787826420.123867:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826420.123892:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332722176:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826420.131429:0:2393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131469:0:2393:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131487:0:2393:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131500:0:2393:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787826420.131520:0:2393:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826420.131534:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787826420.131556:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131565:0:2393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131573:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826420.131583:0:2393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787826420.131611:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332722176:192.168.203.156@tcp:101:rm.0 00000100:00100000:3.0:1787826420.143570:0:2387:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826420.143644:0:6158:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787826420.143669:0:6158:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346388864 00000100:00100000:3.0:1787826420.143683:0:6158:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346388864 00000100:00100000:3.0:1787826420.143764:0:6158:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43711 00000100:00100000:3.0:1787826420.143777:0:6158:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057c6d89850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:17813:x1874663346388864:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826420.143800:0:6158:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9e00/0x95194e8760801d8d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xe21309aa81f4cca5 expref: -99 pid: 229246 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826420.143888:0:6158:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826420.143896:0:6158:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057c6d89850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:17813:x1874663346388864:12345-192.168.203.156@tcp:104: Request processed in 122us (337us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826420.143905:0:6158:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43711 00000100:00100000:3.0:1787826420.150451:0:2388:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.156@tcp (source: 12345-192.168.203.156@tcp) 00000100:00100000:3.0:1787826420.150514:0:22705:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346388992 00000100:00100000:3.0:1787826420.150528:0:22705:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346388992 00000100:00100000:3.0:1787826420.150663:0:22705:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43712 00000100:00100000:3.0:1787826420.150677:0:22705:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057fb223480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:17813:x1874663346388992:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826420.150700:0:22705:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826420.150882:0:22705:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826420.150891:0:22705:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057fb223480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:fc0b407b-a541-420d-8712-b9c8da18fe8c+5:17813:x1874663346388992:12345-192.168.203.156@tcp:104: Request processed in 216us (536us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826420.150904:0:22705:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43712 00010000:00010000:0.0:1787826420.151439:0:34880:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826420.151452:0:34880:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151465:0:34880:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9058021f9200 already unused, calling callback (ffffffffc19e73c0) 00010000:00010000:0.0:1787826420.151470:0:34880:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151481:0:34880:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151497:0:34880:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826420.151529:0:34880:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6ea:0x0]: losing layout lock 00010000:00010000:0.0:1787826420.151614:0:34880:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151623:0:34880:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826420.151746:0:34880:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151755:0:34880:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9058021f9200/0x95194e8760801d94 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6ea:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xe21309aa81f4ccc8 expref: -99 pid: 229246 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826420.151780:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826420.151782:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826420.151783:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826420.151793:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1787826420.151813:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332722688:192.168.203.156@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787826420.155764:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332722688:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826420.518352:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826421.549604:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826421.953999:0:6159:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874663346390912 00000100:00100000:3.0:1787826421.954013:0:6159:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874663346390912 00000100:00100000:3.0:1787826421.954113:0:6159:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.156@tcp, seq: 43713 00000100:00100000:3.0:1787826421.954124:0:6159:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:20167:x1874663346390912:12345-192.168.203.156@tcp:104: 00010000:00010000:3.0:1787826421.954148:0:6159:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787826421.954333:0:6159:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787826421.954342:0:6159:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9057d0de2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:20167:x1874663346390912:12345-192.168.203.156@tcp:104: Request processed in 219us (467us total) trans 0 rc 0/0 00000100:00100000:3.0:1787826421.954351:0:6159:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.156@tcp, seq: 43713 00010000:00010000:0.0:1787826421.954403:0:6634:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787826421.954413:0:6634:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826421.954425:0:6634:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9057d86d2600 already unused, calling callback (ffffffffc10b0300) 00010000:00010000:0.0:1787826421.954434:0:6634:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826421.954491:0:6634:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826421.954500:0:6634:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787826421.954785:0:6634:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826421.954795:0:6634:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9057d128a800 lock: 000000009d7e1073/0x95194e8760801da9 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x525:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xe21309aa81f4ccf9 expref: -99 pid: 229287 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826421.954824:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826421.954827:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826421.954829:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826421.954850:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332724224:192.168.203.156@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787826422.564350:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826425.636989:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1787826426.438456:0:229853:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:27:05 (1787826425) 00010000:00010000:0.0:1787826428.706412:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826428.910942:0:229971:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826428.912215:0:229971:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826428.912304:0:229971:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801dbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 229971 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826428.912315:0:229971:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801dbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229971 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826428.912325:0:229971:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f8400/0x95194e8760801dbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229971 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826428.912351:0:229971:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:229971:1874663332725248:192.168.203.156@tcp:101:touch.0 00000100:00100000:3.0:1787826428.913096:0:229971:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9cdf3c0 going to sleep for 11 seconds 00000080:00200000:3.0:1787826428.976003:0:229971:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6eb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1787826428.976025:0:229971:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f46d.sanityn-2:[0x200000402:0x6eb:0x0](ffff9057dfdd2a08) current size 0, valid attrs 100201f0, mode 9057, uid -1052197296, gid -19182, new size 18446744072537700887, atime 1787826428.000000000, mtime 1787826428.000000000, ctime 1787826428.000000000, rc 0 00000080:00200008:3.0:1787826428.976057:0:229971:0:(file.c:479:ll_file_release()) START file f46d.sanityn-2:[0x200000402:0x6eb:0x0](ffff9057dfdd2a08), flags 104001 00000002:00100000:3.0:1787826428.976182:0:229971:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9057d0de2d80 x1874663332725504/t8589957530(8589957530) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/664 e 0 to 0 dl 1787826444 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787826428.977947:0:229971:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057d128a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787826428.977964:0:229971:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057fb221f80 x1874663332726272/t0(0) o35->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787826428.977994:0:229971:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221f80 pname:cluuid:pid:xid:nid:opc:job touch:398e0287-d3c7-44dd-b1d9-6b85f698e615:229971:1874663332726272:192.168.203.156@tcp:35:touch.0 00000100:00100000:3.0:1787826428.978089:0:229971:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d9cdf3c0 going to sleep for 11 seconds 00010000:00010000:0.0:1787826429.730523:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826430.424964:0:230034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0380 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:230034:1874663332726400:192.168.203.156@tcp:101:link.0 00010000:00010000:0.0:1787826430.425001:0:230034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425017:0:230034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425037:0:230034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425051:0:230034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425066:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425076:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.425085:0:230034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826430.426416:0:230034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057d0de0380 x1874663332726400/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826446 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826430.426445:0:230034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.426463:0:230034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826430.426471:0:230034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826430.426512:0:230034:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826430.426519:0:230034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801dd3 00000080:00010000:0.0:1787826430.426526:0:230034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826430.426529:0:230034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801dd3 from it ffffb512c2047bd8 00010000:00010000:0.0:1787826430.426536:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.426547:0:230034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826430.426585:0:230034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826430.426608:0:230034:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46d.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=8 00000080:00200000:0.0:1787826430.426613:0:230034:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46d.sanityn-2, dir=[0x200000007:0x1:0x0](ffff9057c9129988), intent=getattr 00000002:00010000:0.0:1787826430.426641:0:230034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46d.sanityn-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826430.426708:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 230034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.426716:0:230034:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 230034 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.426727:0:230034:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 230034 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826430.426755:0:230034:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:230034:1874663332726528:192.168.203.156@tcp:101:link.0 00000100:00100000:0.0:1787826430.426900:0:230034:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2dfe40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826430.431586:0:230034:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job link:398e0287-d3c7-44dd-b1d9-6b85f698e615:230034:1874663332726528:192.168.203.156@tcp:101:link.0 00010000:00010000:0.0:1787826430.431623:0:230034:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431645:0:230034:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431653:0:230034:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431679:0:230034:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431693:0:230034:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431709:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431718:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.431726:0:230034:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826430.431741:0:230034:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332726528/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826446 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'link.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826430.431753:0:230034:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826430.433016:0:230034:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.433046:0:230034:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589935618/1771 (0 0) 00000080:00200000:0.0:1787826430.433054:0:230034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826430.433120:0:230034:0:(namei.c:147:ll_iget()) got inode: [0x200000402:0x6eb:0x0](ffff9057dfdd2a08): rc = 0 00000080:00200000:0.0:1787826430.433898:0:230034:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6eb:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1787826430.433914:0:230034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfdd2a08) for lock 0x95194e8760801dda 00000080:00010000:0.0:1787826430.433931:0:230034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfdd2a08) 00000080:00010000:0.0:1787826430.433935:0:230034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801dda from it ffffb512c2047ba0 00010000:00010000:0.0:1787826430.433942:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.433955:0:230034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057d86d3400/0x95194e8760801dda lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4cdc4 expref: -99 pid: 230034 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826430.434034:0:230034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826430.434056:0:230034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826430.434071:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826430.434084:0:230034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826430.434092:0:230034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826430.434097:0:230034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826430.434100:0:230034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801dd3 00000080:00010000:0.0:1787826430.434104:0:230034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801dd3 from it ffffb512c2047bc0 00010000:00010000:0.0:1787826430.434107:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826430.434114:0:230034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826430.434127:0:230034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826430.435615:0:230034:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46d.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9129988), flags=1536 00000080:00200000:0.0:1787826430.435630:0:230034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826430.435651:0:230034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826430.435683:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826430.435698:0:230034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826430.435706:0:230034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826430.435715:0:230034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826430.435719:0:230034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801dd3 00000080:00010000:0.0:1787826430.435725:0:230034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801dd3 from it ffffb512c2047cc0 00010000:00010000:0.0:1787826430.435729:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826430.435738:0:230034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826430.435751:0:230034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826430.435762:0:230034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6eb:0x0](ffff9057dfdd2a08), inode mode 81a4 mask 6 00000080:00200000:0.0:1787826430.435766:0:230034:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:0.0:1787826430.435772:0:230034:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826430.435779:0:230034:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826430.435793:0:230034:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826430.435799:0:230034:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826430.435802:0:230034:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:0.0:1787826430.435805:0:230034:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801dd3 00000080:00010000:0.0:1787826430.435808:0:230034:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801dd3 from it ffffb512c2047d90 00010000:00010000:0.0:1787826430.435811:0:230034:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826430.435818:0:230034:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9057f9b06400/0x95194e8760801dd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cda8 expref: -99 pid: 230034 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1787826430.435828:0:230034:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 3 00010000:00010000:3.0:1787826430.754818:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787826430.921097:0:230036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826430.921148:0:230036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826430.921256:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.921270:0:230036:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826430.921283:0:230036:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826430.921314:0:230036:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332726912:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826430.921451:0:230036:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2dfe40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826431.394998:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332727168:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.395094:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332727424:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.395123:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332727680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.399955:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332727168:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.399974:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332727424:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.399988:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332727680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826431.407013:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2394:1874663332727040:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826431.778669:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826432.443411:0:230036:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332726912:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826432.443448:0:230036:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443463:0:230036:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443477:0:230036:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443489:0:230036:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443504:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443518:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443526:0:230036:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826432.443540:0:230036:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332726912/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826446 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826432.443555:0:230036:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443571:0:230036:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787826432.443579:0:230036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787826432.443614:0:230036:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:0.0:1787826432.443622:0:230036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801de1 00000080:00010000:0.0:1787826432.443628:0:230036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826432.443631:0:230036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801de1 from it ffffb512c2077b58 00010000:00010000:0.0:1787826432.443635:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443644:0:230036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826432.443688:0:230036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826432.443723:0:230036:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f46d.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), flags=8 00000080:00200000:0.0:1787826432.443729:0:230036:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f46d.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988), intent=getattr 00000002:00010000:0.0:1787826432.443755:0:230036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f46d.sanityn,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787826432.443805:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443813:0:230036:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.443823:0:230036:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 230036 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787826432.443847:0:230036:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332727936:192.168.203.156@tcp:101:rm.0 00000100:00100000:0.0:1787826432.444011:0:230036:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d8082f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787826432.454372:0:230036:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332727936:192.168.203.156@tcp:101:rm.0 00010000:00010000:0.0:1787826432.454409:0:230036:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454436:0:230036:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454445:0:230036:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454458:0:230036:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454471:0:230036:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454488:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454498:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454507:0:230036:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787826432.454521:0:230036:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057f158ed80 x1874663332727936/t0(0) o101->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 584/640 e 0 to 0 dl 1787826448 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787826432.454533:0:230036:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787826432.454548:0:230036:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454561:0:230036:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057c50b0800 type 13 mode 4 res 8589935618/1771 (0 0) 00000080:00200000:0.0:1787826432.454570:0:230036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787826432.454612:0:230036:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6eb:0x0](ffff9057dfe121c8) 00000080:00200000:0.0:1787826432.454737:0:230036:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6eb:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787826432.454751:0:230036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfe121c8) for lock 0x95194e8760801de8 00000080:00010000:0.0:1787826432.454766:0:230036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826432.454769:0:230036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801de8 from it ffffb512c2077b20 00010000:00010000:0.0:1787826432.454773:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.454781:0:230036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1787826432.454835:0:230036:0:(file.c:6296:ll_getattr_dentry()) START file f46d.sanityn:[0x200000402:0x6eb:0x0](ffff9057dfe121c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787826432.454842:0:230036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6eb:0x0](ffff9057dfe121c8),name=f46d.sanityn 00000002:00010000:0.0:1787826432.454868:0:230036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6eb:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826432.454878:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826432.454889:0:230036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826432.454896:0:230036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826432.454901:0:230036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfe121c8) 00000080:00010000:0.0:1787826432.454905:0:230036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6eb:0x0](ffff9057dfe121c8) for lock 0x95194e8760801de8 00000080:00010000:0.0:1787826432.454909:0:230036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801de8 from it ffffb512c2077cb8 00010000:00010000:0.0:1787826432.454912:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826432.454920:0:230036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826432.454940:0:230036:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6eb:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787826432.454951:0:230036:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000402:0x6eb:0x0] 00000008:00010000:0.0:1787826432.454970:0:230036:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787826432.454974:0:230036:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9057ede5d1b0, osc lock ffff9057c8b82d10, flags 0x400001002 00000100:00080000:0.0:1787826432.454997:0:230036:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 6491 00000100:00080000:0.0:1787826432.455009:0:230036:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826432.455020:0:230036:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 484638720, lost: 0. 00000100:00100000:0.0:1787826432.455415:0:230036:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c380 x1874663332728064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826432.455646:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801def lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x526:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 230036 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787826432.455671:0:230036:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801def lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x526:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 230036 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787826432.455717:0:230036:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00080000:0.0:1787826432.455745:0:2393:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9057f158ed80 x1874663332728192/t0(0) o101->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826432.459266:0:2392:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9057c7539000: committing for last_committed 4294976174 gen 20 00000100:00080000:0.0:1787826432.459291:0:2392:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9057c7539000: connect to target with instance 1 00000100:00080000:0.0:1787826432.459304:0:2392:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9057c7539000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1787826432.459321:0:2392:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1787826432.459326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1787826432.459338:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1787826432.459345:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1787826432.459363:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000003169dadf): req@ffff9057f158ed80 x1874663332728192/t0(0) o101->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787826432.459391:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332728064:192.168.203.156@tcp:8:rm.0 00000080:00200000:0.0:1787826432.463958:0:230036:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6eb:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787826432.463966:0:230036:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9057c2f89720 (ffff9057d889c2d8/0) at cl_lock_release():245 00000020:00010000:0.0:1787826432.463970:0:230036:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057c2f89720 (ffff9057d889c2d8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787826432.463976:0:230036:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9057ede5d1b0 (ffff9057d889c998/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787826432.463982:0:230036:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057c2f89720 (ffff9057d889c2d8/0) at cl_lock_fini():72 00000020:00010000:0.0:1787826432.463986:0:230036:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9057ede5d1b0 (ffff9057d889c2d8/1) at cl_lock_fini():72 00000080:00200000:0.0:1787826432.464003:0:230036:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6eb:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787826432.464159:0:230036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826432.464190:0:230036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826432.464210:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826432.464251:0:230036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787826432.464333:0:230036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826432.464343:0:230036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826432.464347:0:230036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801de1 00000080:00010000:0.0:1787826432.464354:0:230036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801de1 from it ffffb512c2077c10 00010000:00010000:0.0:1787826432.464358:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787826432.464366:0:230036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826432.464378:0:230036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 1 00000080:00200000:0.0:1787826432.464391:0:230036:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f46d.sanityn, flags=0 00000080:00200000:0.0:1787826432.464398:0:230036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988),name=/ 00000002:00010000:0.0:1787826432.464406:0:230036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787826432.464413:0:230036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826432.464422:0:230036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826432.464430:0:230036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787826432.464434:0:230036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) 00000080:00010000:0.0:1787826432.464437:0:230036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9199988) for lock 0x95194e8760801de1 00000080:00010000:0.0:1787826432.464441:0:230036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801de1 from it ffffb512c2077d68 00010000:00010000:0.0:1787826432.464444:0:230036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826432.464452:0:230036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787826432.464459:0:230036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9199988), inode mode 41ff mask 3 00000080:00200000:0.0:1787826432.464465:0:230036:0:(namei.c:2418:ll_unlink()) VFS Op:name=f46d.sanityn, dir=[0x200000007:0x1:0x0](ffff9057c9199988) 00010000:00010000:0.0:1787826432.464495:0:230036:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826432.464524:0:230036:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787826432.464535:0:230036:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6eb:0x0]: losing layout lock 00010000:00010000:0.0:1787826432.464575:0:230036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826432.464583:0:230036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787826432.464589:0:230036:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787826432.464592:0:230036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d3600/0x95194e8760801de8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe21309aa81f4ce03 expref: -99 pid: 230036 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826432.464614:0:230036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9057c7539000 lock: ffff9057d86d2c00/0x95194e8760801de1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe21309aa81f4cde0 expref: -99 pid: 230036 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787826432.464634:0:230036:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9057c7539000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787826432.464643:0:230036:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9057f158c380 x1874663332728448/t0(0) o36->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787826432.464685:0:230036:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332728448:192.168.203.156@tcp:36:rm.0 00000100:00100000:0.0:1787826432.464791:0:230036:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057ef2dfe40 going to sleep for 11 seconds 00000100:00100000:0.0:1787826432.470185:0:230036:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c380 pname:cluuid:pid:xid:nid:opc:job rm:fc0b407b-a541-420d-8712-b9c8da18fe8c:230036:1874663332728448:192.168.203.156@tcp:36:rm.0 00010000:00010000:0.0:1787826432.802463:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787826433.010981:0:229720:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988),name=/ 00000002:00010000:3.0:1787826433.011018:0:229720:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787826433.011099:0:229720:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 229720 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.011111:0:229720:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229720 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.011121:0:229720:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 229720 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787826433.011152:0:229720:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:229720:1874663332728576:192.168.203.156@tcp:101:bash.0 00000100:00100000:3.0:1787826433.011305:0:229720:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d8b0b240 going to sleep for 11 seconds 00000100:00100000:3.0:1787826433.014670:0:229720:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:229720:1874663332728576:192.168.203.156@tcp:101:bash.0 00010000:00010000:3.0:1787826433.014705:0:229720:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014720:0:229720:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014734:0:229720:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014746:0:229720:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014761:0:229720:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014770:0:229720:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014778:0:229720:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787826433.014791:0:229720:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9057fb220a80 x1874663332728576/t0(0) o101->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 576/688 e 0 to 0 dl 1787826449 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787826433.014806:0:229720:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014820:0:229720:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9057d1282400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1787826433.014828:0:229720:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1787826433.014863:0:229720:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 73728 00000080:00010000:3.0:1787826433.014871:0:229720:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) for lock 0x95194e8760801df6 00000080:00010000:3.0:1787826433.014877:0:229720:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9057c9129988) 00000080:00010000:3.0:1787826433.014881:0:229720:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x95194e8760801df6 from it ffffb512c0da3bb0 00010000:00010000:3.0:1787826433.014885:0:229720:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787826433.014894:0:229720:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787826433.014923:0:229720:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9057c9129988), inode mode 41ff mask 44 00000080:00200008:3.0:1787826433.014938:0:229720:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), flags 304000 00000080:00200008:3.0:1787826433.014972:0:229720:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9057c9129988), request_mask 2047, flags 0, foreign 0 00000002:00010000:3.0:1787826433.015000:0:229720:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: readdir flags 0 00010000:00010000:3.0:1787826433.015161:0:229720:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826433.015175:0:229720:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787826433.015885:0:229720:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:229720:1874663332728704:192.168.203.156@tcp:37:bash.0 00000100:00100000:3.0:1787826433.015997:0:229720:0:(client.c:2715:ptlrpc_set_wait()) set ffff9057d8b0b240 going to sleep for 11 seconds 00000100:00100000:3.0:1787826433.034980:0:229720:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job bash:398e0287-d3c7-44dd-b1d9-6b85f698e615:229720:1874663332728704:192.168.203.156@tcp:37:bash.0 00010000:00010000:3.0:1787826433.035920:0:229720:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787826433.035936:0:229720:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9057d128a800 lock: ffff9058021f9600/0x95194e8760801df6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xe21309aa81f4ce34 expref: -99 pid: 229720 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787826435.875008:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826436.514504:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332729344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514615:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729600:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514657:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729856:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514681:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332730112:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514710:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:0.0:1787826436.514716:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729472:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514735:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729728:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514755:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332729984:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826436.514774:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332730240:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826438.946468:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826439.970475:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826440.994557:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787826441.634528:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730496:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634655:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730752:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634679:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731008:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634703:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731264:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634737:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:3.0:1787826441.634746:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2396:1874663332730368:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634779:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730624:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.634809:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826441.636182:0:2396:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731136:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826442.018468:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826443.042454:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826446.114434:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826446.754485:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332731520:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826446.754741:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332731776:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826446.754782:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332732032:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826446.754819:0:2394:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332732288:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826449.186431:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826450.210833:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826451.234366:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787826451.874445:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332732416:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826451.874856:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332732672:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826451.874898:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb239880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332732928:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826451.874934:0:2393:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb23b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332733184:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826452.258458:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787826452.898346:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb23bb80 x1874663332729472/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.908753:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb23bb80 x1874663332729472/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.908766:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.908771:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9057d128a800: Connection to lustre-MDT0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.914670:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.914749:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb238700 x1874663332729728/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.914782:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb238700 x1874663332729728/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.914795:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.914801:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9057d128a800: Connection to lustre-OST0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.914806:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.914827:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb239180 x1874663332729984/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.914845:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb239180 x1874663332729984/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.914856:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.914860:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9057c7539000: Connection to lustre-MDT0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.914863:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.914879:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb23ad80 x1874663332730240/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.914905:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb23ad80 x1874663332730240/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.914918:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.914925:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9057c7539000: Connection to lustre-OST0001 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.914932:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.914952:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb238e00 x1874663332729344/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.914987:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb238e00 x1874663332729344/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.915001:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1787826452.915010:0:2393:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.156@tcp: Connection to MGS (at 192.168.203.156@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1787826452.919583:0:2393:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.156@tcp for MGC192.168.203.156@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1787826452.919586:0:2393:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1787826452.919594:0:2393:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9057fb238e00 x1874663332729344/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919605:0:2393:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9057fb220a80 x1874663332730368/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826457 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919620:0:2393:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9057f158fb80 x1874663332731392/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826462 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919628:0:2393:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff9057fb238a80 x1874663332732416/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1787826467 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1787826452.919643:0:2393:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1787826452.919647:0:2393:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1787826452.919648:0:2393:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1787826452.919651:0:2393:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.919675:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb239c00 x1874663332729600/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919755:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb239c00 x1874663332729600/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.919771:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.919776:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9057d128a800: Connection to lustre-MDT0001 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.919780:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.919787:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb238380 x1874663332729856/t0(0) o400->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919796:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb238380 x1874663332729856/t0(0) o400->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.919801:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.919805:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9057d128a800: Connection to lustre-OST0001 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.919809:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1787826452.919817:0:2393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826436/real 1787826436] req@ffff9057fb23b100 x1874663332730112/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826452.919832:0:2393:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb23b100 x1874663332730112/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826452 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826452.919839:0:2393:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787826452.919841:0:2393:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9057c7539000: Connection to lustre-MDT0001 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1787826452.919844:0:2393:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1787826452.919862:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332732416:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919884:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332731392:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919900:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729472:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919905:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729728:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919912:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332729984:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919917:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332730240:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919926:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2393:1874663332729344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919934:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729600:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919939:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:398e0287-d3c7-44dd-b1d9-6b85f698e615:2393:1874663332729856:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826452.919942:0:2393:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc0b407b-a541-420d-8712-b9c8da18fe8c:2393:1874663332730112:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826453.282420:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826456.354439:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1787826458.018665:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb221500 x1874663332730624/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038403:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb221500 x1874663332730624/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038419:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c7538800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787826458.038428:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb220e00 x1874663332730880/t0(0) o400->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038445:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb220e00 x1874663332730880/t0(0) o400->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038452:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c753a000 already not connected (conn 40, was 39): CONNECTING 00000100:00000400:3.0:1787826458.038459:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb222680 x1874663332731136/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038485:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb222680 x1874663332731136/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038493:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c83da800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787826458.038499:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb223800 x1874663332730496/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038510:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb223800 x1874663332730496/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038516:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c752b800 already not connected (conn 5, was 4): CONNECTING 00000100:00000400:3.0:1787826458.038521:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb221180 x1874663332730752/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038531:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb221180 x1874663332730752/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038537:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c753c800 already not connected (conn 34, was 33): CONNECTING 00000100:00000400:3.0:1787826458.038542:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb223b80 x1874663332731008/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038551:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb223b80 x1874663332731008/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038558:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c83db000 already not connected (conn 5, was 4): CONNECTING 00000100:00000400:3.0:1787826458.038563:0:2396:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826441/real 1787826441] req@ffff9057fb220000 x1874663332731264/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787826458.038573:0:2396:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb220000 x1874663332731264/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826457 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826458.038578:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c83d9800 already not connected (conn 21, was 20): CONNECTING 00000100:00100000:3.0:1787826458.038592:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730624:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038598:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038603:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731136:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038607:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730496:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038612:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2396:1874663332730752:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038616:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731008:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1787826458.038620:0:2396:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2396:1874663332731264:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826459.426517:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826460.450423:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826461.474483:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826462.117124:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117129:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826462.117131:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117137:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117145:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117151:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826462.117178:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332734976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117210:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117214:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826462.117215:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117234:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117237:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117241:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117249:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826462.117259:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332735104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117267:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117270:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826462.117271:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117274:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117277:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117280:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117284:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826462.117292:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332735232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117300:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117302:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826462.117304:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117306:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117310:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117313:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117317:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826462.117324:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332735360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117331:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117334:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826462.117335:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117339:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117341:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117344:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117348:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826462.117355:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332735488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117362:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826462.117366:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117369:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117372:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117375:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117379:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826462.117386:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332735616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117397:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117400:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826462.117402:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117404:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117408:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117411:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117414:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826462.117421:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332735744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826462.117429:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826462.117431:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826462.117433:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826462.117436:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8268 00000100:00080000:3.0:1787826462.117438:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787826462.117441:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826462.117455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826462.117464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332735872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000400:0.0:1787826462.119877:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826446/real 1787826446] req@ffff9057f158d180 x1874663332731520/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826462.119900:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057f158d180 x1874663332731520/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826462.119908:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c752b800 already not connected (conn 6, was 4): CONNECTING 00000100:00000400:0.0:1787826462.119914:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826446/real 1787826446] req@ffff9057f158ce00 x1874663332731776/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826462.119924:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057f158ce00 x1874663332731776/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826462.119936:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c753c800 already not connected (conn 35, was 33): CONNECTING 00000100:00000400:0.0:1787826462.119941:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826446/real 1787826446] req@ffff9057f158e300 x1874663332732032/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826462.119972:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057f158e300 x1874663332732032/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826462.119979:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c83db000 already not connected (conn 6, was 4): CONNECTING 00000100:00000400:0.0:1787826462.119984:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826446/real 1787826446] req@ffff9057f158f100 x1874663332732288/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826462.119994:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057f158f100 x1874663332732288/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826462 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826462.120001:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c83d9800 already not connected (conn 22, was 20): CONNECTING 00000100:00100000:0.0:1787826462.120014:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332731520:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826462.120020:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332731776:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826462.120025:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332732032:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826462.120029:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332732288:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826462.498451:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826463.522500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826466.594493:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787826467.234514:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9057fb23aa00 x1874663332732544/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787826467.234714:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234720:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234723:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234729:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234732:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234735:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234738:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234740:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234742:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234745:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234747:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234750:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234752:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234755:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826467.234758:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826467.234761:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787826467.254372:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb23aa00 x1874663332732544/t0(0) o400->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826467.254385:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c752b800 already not connected (conn 6, was 4): CONNECTING 00000100:00000400:0.0:1787826467.254396:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9057fb238000 x1874663332732800/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826467.254423:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb238000 x1874663332732800/t0(0) o400->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826467.254431:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c753c800 already not connected (conn 35, was 33): CONNECTING 00000100:00000400:0.0:1787826467.254438:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9057fb239500 x1874663332733056/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826467.254450:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb239500 x1874663332733056/t0(0) o400->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826467.254457:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c83db000 already not connected (conn 6, was 4): CONNECTING 00000100:00000400:0.0:1787826467.254463:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826451/real 1787826451] req@ffff9057fb23b480 x1874663332733312/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826467.254474:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057fb23b480 x1874663332733312/t0(0) o400->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1787826467 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826467.254479:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9057c83d9800 already not connected (conn 22, was 20): CONNECTING 00000100:00100000:0.0:1787826467.254495:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332732544:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826467.254502:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb238000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332732800:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826467.254506:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb239500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332733056:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826467.254511:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb23b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332733312:192.168.203.156@tcp:400:kworker.0 00000100:00000400:0.0:1787826468.258355:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826452/real 1787826452] req@ffff9057c78a8700 x1874663332733696/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826468.258392:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057c78a8700 x1874663332733696/t0(0) o400->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826468 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826468.258409:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c7538800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1787826468.258417:0:2394:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826452/real 1787826452] req@ffff9057c78a9180 x1874663332734208/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787826468.258429:0:2394:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9057c78a9180 x1874663332734208/t0(0) o400->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1787826468 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787826468.258434:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9057c83da800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1787826468.258441:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:398e0287-d3c7-44dd-b1d9-6b85f698e615:2394:1874663332733696:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1787826468.258445:0:2394:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc0b407b-a541-420d-8712-b9c8da18fe8c:2394:1874663332734208:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1787826469.666429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826470.690416:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826471.714661:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826472.738455:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826473.762486:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826476.834495:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826477.474432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474445:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474459:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474468:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474472:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474481:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474489:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474495:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474498:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474501:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474506:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826477.474851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474855:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826477.474858:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.474864:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.474872:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.474878:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826477.474907:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332736000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.474929:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474932:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826477.474934:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.474938:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.474941:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.474945:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.474954:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826477.474966:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332736128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.474974:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.474977:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826477.474978:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.474981:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.474985:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.474988:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.474992:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826477.475000:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332736256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.475007:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.475009:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826477.475011:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.475014:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.475016:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.475019:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.475023:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826477.475030:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332736384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.475036:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.475038:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826477.475039:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.475042:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.475045:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.475048:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.475051:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826477.475058:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332736512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.475066:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.475069:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826477.475070:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.475074:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.475078:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.475081:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.475085:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826477.475093:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332736640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.475101:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.475104:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826477.475106:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.475109:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.475111:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.475115:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.475119:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826477.475127:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332736768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826477.475135:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826477.475138:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826477.475139:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826477.475142:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8277 00000100:00080000:3.0:1787826477.475144:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787826477.475148:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826477.475151:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826477.475168:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332736896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826479.906521:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826480.933642:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826481.954537:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826482.594798:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594821:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594824:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594827:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594829:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594832:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594834:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594837:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594840:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594843:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594845:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594848:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594853:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826482.594856:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826482.594859:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1787826482.978651:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826484.002592:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826487.074644:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826487.714379:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714388:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714391:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714394:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714396:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714399:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714401:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714405:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714410:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714412:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714416:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714418:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714421:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826487.714424:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826487.714427:0:149405:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1787826490.150124:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826491.170803:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826492.194736:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826492.834392:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834401:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834403:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834406:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834408:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834410:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834413:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834415:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834417:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834420:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834423:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834425:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834428:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834431:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787826492.834434:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826492.834437:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1787826493.219768:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787826493.730332:0:2384:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.203.156@tcp (192.168.203.156:988) timed out; the network or node may be down. 00010000:00010000:0.0:1787826494.242467:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826497.314481:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826497.954321:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954332:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954337:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954342:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954344:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954347:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954350:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954354:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954356:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954361:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954365:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954372:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954376:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954380:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954383:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954388:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954570:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954575:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954581:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954584:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954591:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954595:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954600:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954609:0:229536:0:(pinger.c:247:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787826497.954614:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954619:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826497.954623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954632:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954644:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954653:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954668:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826497.954701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332737024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954720:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954724:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826497.954729:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954733:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954736:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954742:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954750:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826497.954765:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332737152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954775:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954790:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826497.954794:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954799:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954802:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954806:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954811:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826497.954823:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332737280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954835:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954837:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826497.954839:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954842:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954845:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954848:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826497.954858:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332737408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954865:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954868:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826497.954870:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954875:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954880:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954887:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954892:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826497.954904:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332737536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954912:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954915:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826497.954916:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954922:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954930:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954934:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954946:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826497.954959:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332737664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.954970:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.954973:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826497.954975:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.954979:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.954983:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.954988:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.954995:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826497.955008:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332737792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826497.955017:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826497.955022:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826497.955024:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826497.955029:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8293 00000100:00080000:0.0:1787826497.955034:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1787826497.955041:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826497.955046:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826497.955057:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332737920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:3.0:1787826499.362731:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.156:988 00000400:00000100:3.0:1787826499.362773:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826499.362789:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362886:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362896:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362904:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362910:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362917:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362929:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826499.362936:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787826500.386503:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826501.410547:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826502.434879:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826503.074326:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074345:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826503.074349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074358:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074371:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074379:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826503.074411:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332738048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074430:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074433:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826503.074434:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074437:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074440:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074449:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826503.074458:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332738176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074465:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074467:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826503.074468:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074470:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074471:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074476:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826503.074489:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332738304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826503.074502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074505:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074509:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074513:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826503.074530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332738432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074540:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074543:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826503.074544:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074547:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074551:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074554:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074559:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826503.074571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332738560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826503.074589:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074596:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074600:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074604:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074609:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826503.074620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332738688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826503.074637:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074643:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074651:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074657:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826503.074671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332738816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826503.074681:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826503.074684:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826503.074686:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826503.074692:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8313 00000100:00080000:3.0:1787826503.074697:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826503.074702:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826503.074708:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826503.074723:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332738944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826503.458444:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826504.482762:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826505.506573:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.156:988 00000400:00000100:3.0:1787826505.506621:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826505.506638:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506680:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506688:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506696:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506706:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506716:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506727:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826505.506740:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787826507.554440:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826508.194333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826508.194347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194395:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826508.194435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332739072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826508.194468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194476:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826508.194502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332739200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826508.194526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194530:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194533:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194538:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826508.194555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332739328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826508.194569:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194573:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194576:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194580:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826508.194596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332739456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826508.194610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194613:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194616:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194621:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194626:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826508.194636:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332739584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194645:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826508.194652:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194661:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194664:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194668:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194672:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826508.194685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332739712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194705:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194709:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826508.194711:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194716:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194720:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194726:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826508.194763:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332739840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826508.194774:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826508.194778:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826508.194781:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826508.194788:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8318 00000100:00080000:3.0:1787826508.194794:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826508.194799:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826508.194805:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826508.194821:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332739968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:3.0:1787826508.580786:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.156:988 00000400:00000100:3.0:1787826508.580844:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826508.580861:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580902:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580911:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580918:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580926:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580934:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580943:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826508.580958:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787826510.627771:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826511.650551:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826512.674514:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826513.314339:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314350:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826513.314354:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314364:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314376:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314383:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826513.314420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332740096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314456:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314460:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826513.314462:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314470:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314473:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826513.314496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332740224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314505:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826513.314510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314513:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314516:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314520:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314524:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826513.314535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332740352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314554:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826513.314563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314567:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314570:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314652:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826513.314670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332740480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314679:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314682:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826513.314684:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314689:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314692:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314695:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314700:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826513.314710:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332740608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314719:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314723:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826513.314725:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314729:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314732:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314736:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314741:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826513.314751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332740736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314760:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314764:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826513.314765:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314770:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314773:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314776:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314781:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826513.314790:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332740864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826513.314800:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826513.314803:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826513.314805:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826513.314809:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8324 00000100:00080000:3.0:1787826513.314812:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826513.314815:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826513.314819:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826513.314829:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332740992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826513.698482:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826514.722660:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826517.797067:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826518.434337:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434347:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826518.434351:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434361:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434371:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434377:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826518.434410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332741120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434447:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434450:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826518.434452:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434456:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434460:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434463:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434471:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826518.434483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332741248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826518.434496:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434499:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434502:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434505:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826518.434516:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332741376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826518.434527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434531:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434534:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434537:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826518.434550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332741504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434559:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826518.434561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434565:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434571:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434574:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826518.434582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332741632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826518.434593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434597:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434599:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434602:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434606:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826518.434613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332741760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434624:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434627:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826518.434628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434631:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434633:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434635:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826518.434645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332741888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826518.434652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826518.434654:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826518.434656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826518.434658:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8329 00000100:00080000:3.0:1787826518.434661:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826518.434664:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826518.434667:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826518.434674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332742016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826521.890469:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826522.914835:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826523.554341:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.554350:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826523.554354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.554364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.554373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.554379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826523.554414:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332742144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.554696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.554700:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826523.554702:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.554707:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.554711:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.554716:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.554724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826523.554739:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332742272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.554964:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.554968:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826523.554970:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.554975:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.554979:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.554982:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.554987:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826523.554999:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332742400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.555088:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.555092:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826523.555093:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.555097:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.555101:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.555104:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.555109:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826523.555120:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332742528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.555210:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.555214:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826523.555310:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.555315:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.555320:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.555323:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.555328:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826523.555340:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332742656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.555356:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.555360:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826523.555361:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.555365:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.555368:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.555372:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.555377:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826523.555387:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332742784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.555405:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.555408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826523.555410:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.555414:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.555419:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.555422:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.555427:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826523.555505:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332742912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826523.555523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826523.555526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826523.555529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826523.555533:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8334 00000100:00080000:3.0:1787826523.555537:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826523.555544:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826523.555554:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826523.555569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332743040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826524.962510:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826528.034956:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826528.674348:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674368:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826528.674372:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674382:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674392:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674398:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826528.674432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332743168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674456:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826528.674458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826528.674482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332743296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826528.674492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674503:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674506:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674509:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826528.674516:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332743424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826528.674525:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674527:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674529:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674531:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674533:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826528.674539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332743552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826528.674548:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674549:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674558:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826528.674572:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332743680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674581:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826528.674582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674585:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674588:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674590:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826528.674598:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332743808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826528.674607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674609:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674610:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674612:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826528.674621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332743936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826528.674629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826528.674639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826528.674641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826528.674646:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8339 00000100:00080000:3.0:1787826528.674650:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826528.674655:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826528.674661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826528.674673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332744064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826532.130446:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826533.154542:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826533.794276:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794284:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826533.794287:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794295:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794303:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794308:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826533.794347:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332744192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794368:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794372:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826533.794374:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794378:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794382:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794386:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794394:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826533.794427:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332744320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826533.794443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794451:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826533.794464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332744448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794471:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826533.794472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794475:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794477:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794478:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794481:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826533.794489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332744576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826533.794499:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794502:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794506:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794508:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826533.794517:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332744704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826533.794525:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794527:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794529:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794531:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826533.794542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332744832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826533.794565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794570:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794578:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794589:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826533.794597:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332744960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826533.794611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826533.794613:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826533.794615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826533.794618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8344 00000100:00080000:3.0:1787826533.794620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826533.794622:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826533.794625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826533.794633:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332745088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826535.202644:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826538.274506:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826538.914302:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914311:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826538.914316:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914324:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914334:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826538.914379:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332745216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914410:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914413:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826538.914415:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914418:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914421:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914425:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914432:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826538.914445:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332745344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826538.914464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914468:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914471:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914474:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826538.914490:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332745472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914498:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914501:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826538.914503:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914506:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826538.914524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332745600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914533:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826538.914535:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914538:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914541:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914544:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914547:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826538.914554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332745728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826538.914565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914569:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914575:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826538.914585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332745856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914596:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826538.914597:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914603:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914606:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826538.914618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332745984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826538.914630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826538.914633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826538.914634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826538.914637:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8349 00000100:00080000:3.0:1787826538.914639:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826538.914642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826538.914645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826538.914653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332746112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826542.370942:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826543.394555:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826544.034334:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826544.034347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034376:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826544.034416:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332746240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034445:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034449:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826544.034450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034455:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034462:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034472:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826544.034498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332746368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826544.034514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034521:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034524:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826544.034550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332746496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034558:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826544.034563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034567:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034570:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034575:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826544.034605:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332746624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034625:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826544.034627:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034633:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034644:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826544.034662:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332746752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034674:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826544.034684:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034690:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034697:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034703:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034709:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826544.034722:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332746880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034734:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826544.034741:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034745:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034749:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034756:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034763:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826544.034776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332747008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826544.034789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826544.034793:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826544.034795:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826544.034798:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8354 00000100:00080000:3.0:1787826544.034802:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826544.034805:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826544.034811:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826544.034823:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332747136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:3.0:1787826545.442680:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:0.0:1787826545.442698:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826545.442730:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826545.442748:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442777:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442783:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442788:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442793:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442798:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442804:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826545.442809:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:3.0:1787826548.514638:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826549.154326:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154333:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826549.154336:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154344:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154357:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826549.154386:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332747264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154417:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154421:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826549.154422:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154426:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154430:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154442:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826549.154459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332747392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154467:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154470:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826549.154473:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154479:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154483:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154487:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154499:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826549.154516:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332747520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154525:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154528:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826549.154530:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154533:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154538:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154543:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826549.154559:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332747648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154565:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154567:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826549.154569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154571:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154574:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154575:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154578:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826549.154585:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332747776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154590:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154592:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826549.154594:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154598:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154602:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154607:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154612:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826549.154622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332747904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154633:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154638:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826549.154639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154650:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154653:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826549.154666:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332748032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826549.154674:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826549.154680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826549.154683:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826549.154686:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8359 00000100:00080000:0.0:1787826549.154688:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826549.154690:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826549.154693:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826549.154698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332748160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826552.611525:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826553.634553:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826554.274339:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274350:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826554.274353:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274363:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826554.274410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332748288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826554.274454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274458:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826554.274483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332748416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826554.274492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274494:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274496:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274499:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826554.274507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332748544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274512:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826554.274515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274517:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274518:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274520:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826554.274527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332748672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826554.274533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274535:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274538:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274540:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826554.274545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332748800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826554.274554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274558:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274561:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274565:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826554.274578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332748928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274586:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826554.274591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274594:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274597:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274601:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826554.274614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332749056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826554.274622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826554.274625:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826554.274627:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826554.274629:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8365 00000100:00080000:3.0:1787826554.274633:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826554.274636:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826554.274641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826554.274650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332749184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826555.682398:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826558.754828:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826559.394335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394344:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826559.394348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394358:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826559.394406:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332749312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826559.394434:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394437:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394440:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394443:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826559.394459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332749440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826559.394469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826559.394488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332749568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826559.394508:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394512:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394515:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394519:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826559.394535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332749696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826559.394550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394552:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394555:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826559.394575:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332749824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826559.394585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394587:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394589:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394591:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826559.394602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332749952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826559.394626:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394636:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394643:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826559.394667:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332750080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826559.394681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826559.394685:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826559.394688:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826559.394697:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8370 00000100:00080000:3.0:1787826559.394701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826559.394705:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826559.394709:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826559.394724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332750208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826562.850608:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826563.874882:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826565.922499:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826568.995608:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826573.090584:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826574.114598:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826574.754346:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754356:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826574.754360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754370:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754381:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754392:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826574.754431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332752384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754456:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754459:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826574.754461:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754465:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754468:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754473:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754482:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826574.754497:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332752512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826574.754515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754519:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754522:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754532:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826574.754551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332752640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826574.754567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754578:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754581:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826574.754597:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332752768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826574.754618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754625:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754628:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754635:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754642:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826574.754652:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332752896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754677:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754681:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826574.754685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754691:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754694:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754698:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754702:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826574.754715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332753024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754726:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754729:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826574.754731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754735:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754741:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754748:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754752:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826574.754764:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332753152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826574.754777:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826574.754796:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826574.754799:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826574.754810:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8385 00000100:00080000:3.0:1787826574.754816:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826574.754822:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826574.754829:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826574.754853:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332753280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826576.162578:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826579.234814:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826579.874345:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874355:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826579.874359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874368:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874379:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874386:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826579.874420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332753408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826579.874455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874460:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874463:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874467:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874475:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826579.874488:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332753536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826579.874500:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874503:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874505:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874509:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874513:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826579.874522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332753664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826579.874540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874543:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874545:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874548:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826579.874562:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332753792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874569:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826579.874573:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874576:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874579:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874582:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826579.874595:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332753920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826579.874607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874611:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874614:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874618:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874623:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826579.874645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332754048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874653:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874660:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826579.874662:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874665:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874668:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874672:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826579.874685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332754176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826579.874693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826579.874696:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826579.874697:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826579.874700:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8390 00000100:00080000:0.0:1787826579.874707:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826579.874710:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826579.874714:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826579.874722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332754304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:0.0:1787826582.946565:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787826582.946611:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787826582.946626:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946661:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946671:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946688:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946703:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946716:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946730:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826582.946746:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:3.0:1787826583.332137:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826584.354499:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826584.994687:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994698:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826584.994701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994711:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994720:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994727:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826584.994773:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332754432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.994803:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994807:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826584.994809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994813:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994817:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994821:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.994830:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826584.994842:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332754560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.994851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994854:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826584.994856:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994859:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994862:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994865:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.994869:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826584.994879:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332754688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.994887:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994890:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826584.994891:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994895:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994897:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994901:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.994906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826584.994916:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332754816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.994925:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994928:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826584.994930:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994933:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994936:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994939:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.994943:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826584.994953:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332754944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.994962:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.994965:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826584.994967:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.994971:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.994973:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.994977:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.994986:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826584.994996:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332755072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.995006:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.995021:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826584.995022:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.995026:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.995030:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.995034:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.995038:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826584.995048:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332755200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826584.995057:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826584.995059:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826584.995061:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826584.995064:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8395 00000100:00080000:3.0:1787826584.995066:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826584.995069:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826584.995072:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826584.995080:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332755328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826586.402396:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826589.474401:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826590.114324:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114335:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826590.114339:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114367:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826590.114403:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332755456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114436:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826590.114442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114462:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826590.114472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332755584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114481:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826590.114482:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114484:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114486:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114488:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114490:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826590.114496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332755712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114501:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826590.114504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114507:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826590.114525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332755840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826590.114537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114540:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114543:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114546:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826590.114573:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332755968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826590.114587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114591:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114594:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114597:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826590.114610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332756096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826590.114619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114625:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826590.114632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332756224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826590.114638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826590.114640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826590.114641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826590.114642:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8400 00000100:00080000:3.0:1787826590.114644:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826590.114645:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826590.114647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826590.114651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332756352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826593.570396:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826594.594451:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826595.234335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234351:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826595.234355:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826595.234411:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332756480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234430:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826595.234434:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234437:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234439:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234441:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234448:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826595.234456:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332756608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826595.234465:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234467:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234468:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234472:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826595.234481:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332756736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234499:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826595.234500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234504:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234507:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234511:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826595.234539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332756864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234548:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826595.234557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234563:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234568:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234573:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826595.234594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332756992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826595.234621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234627:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234631:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234639:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826595.234663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332757120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234685:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826595.234688:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234692:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234697:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234704:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234710:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826595.234721:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332757248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826595.234733:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826595.234738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826595.234741:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826595.234750:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8405 00000100:00080000:3.0:1787826595.234755:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826595.234761:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826595.234765:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826595.234776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332757376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826596.642497:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826599.714434:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826600.354686:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354695:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826600.354699:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354708:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354718:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.354724:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826600.354759:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332757504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.354787:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354790:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826600.354792:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354796:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354799:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.354803:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.354811:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826600.354822:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332757632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.354830:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354833:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826600.354835:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354837:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354840:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.354843:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.354846:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826600.354854:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332757760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.354862:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354865:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826600.354866:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354870:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354872:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.354875:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.354878:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826600.354885:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332757888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.354953:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354956:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826600.354957:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354961:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354964:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.354967:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.354971:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826600.354980:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332758016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.354988:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.354991:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826600.354992:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.354995:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.354997:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.355000:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.355003:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826600.355009:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332758144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.355016:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.355019:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826600.355020:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.355022:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.355024:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.355027:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.355031:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826600.355037:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332758272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826600.355044:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826600.355047:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826600.355048:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826600.355051:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8411 00000100:00080000:0.0:1787826600.355061:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826600.355064:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826600.355067:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826600.355074:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332758400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826603.810392:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826604.834412:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826606.882417:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826609.954468:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826614.050431:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826615.074608:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826615.714441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714450:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826615.714454:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714462:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714472:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714479:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826615.714515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332760576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714541:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714544:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826615.714546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714550:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714554:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714558:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826615.714579:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332760704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714591:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826615.714601:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714606:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714609:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714613:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714618:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826615.714629:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332760832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714637:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714641:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826615.714642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714645:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714650:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714653:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826615.714672:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332760960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826615.714687:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714690:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714694:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714699:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714704:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826615.714718:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332761088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714729:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826615.714742:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714749:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714755:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714764:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714771:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826615.714785:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332761216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714797:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714802:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826615.714804:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714809:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714815:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714821:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714828:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826615.714845:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332761344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826615.714860:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826615.714867:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826615.714869:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826615.714878:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8426 00000100:00080000:0.0:1787826615.714884:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826615.714892:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826615.714899:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826615.714916:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332761472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826617.122495:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826620.194495:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826624.290576:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826625.314502:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826625.954333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826625.954343:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826625.954413:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332762624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954443:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826625.954449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954456:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954460:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954468:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826625.954482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332762752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954493:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826625.954500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954504:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954506:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954509:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826625.954521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332762880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826625.954529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954531:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954533:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954535:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954538:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826625.954544:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332763008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954548:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826625.954551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954553:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954556:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954558:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954560:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826625.954566:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332763136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954579:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826625.954580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954583:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954588:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954590:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826625.954603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332763264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954616:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954620:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826625.954624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954639:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954644:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826625.954666:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332763392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826625.954680:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826625.954683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826625.954685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826625.954688:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8436 00000100:00080000:0.0:1787826625.954690:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826625.954693:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826625.954696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826625.954727:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332763520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826627.362553:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826630.434737:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826631.074713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.074723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826631.074726:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.074735:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.074744:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.074751:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826631.074788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332763648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.074947:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.074950:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826631.074952:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.074956:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.074960:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.074964:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.074972:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826631.075001:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332763776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075396:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075400:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826631.075402:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075406:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075410:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075414:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075419:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826631.075431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332763904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075441:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075444:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826631.075446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075449:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075451:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075454:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075458:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826631.075476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332764032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075487:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826631.075489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075492:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075495:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075498:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075502:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826631.075515:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332764160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075531:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826631.075533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075536:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075539:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075542:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826631.075553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332764288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826631.075565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075568:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075570:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075573:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826631.075583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332764416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826631.075592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826631.075595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826631.075596:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826631.075599:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8441 00000100:00080000:3.0:1787826631.075602:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826631.075604:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826631.075608:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826631.075615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332764544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826633.506564:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826634.530416:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826635.556858:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826636.194333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826636.194346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194354:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194364:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194370:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826636.194415:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332764672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826636.194455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194475:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826636.194487:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332764800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826636.194507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194516:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194520:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826636.194527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332764928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826636.194536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194540:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194542:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826636.194549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332765056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194555:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826636.194558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194562:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194568:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826636.194583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332765184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194590:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826636.194594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194597:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194599:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194602:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194606:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826636.194614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332765312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194620:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826636.194624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194627:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826636.194645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332765440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826636.194657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826636.194660:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826636.194661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826636.194677:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8446 00000100:00080000:3.0:1787826636.194680:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826636.194683:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826636.194687:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826636.194696:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332765568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826637.602425:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826640.674496:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826643.746785:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826645.794776:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826647.842450:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826649.506515:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787826649.506558:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826649.506574:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506614:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506624:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506631:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506638:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506644:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506652:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826649.506658:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:3.0:1787826650.914553:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826651.554384:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554399:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826651.554403:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554412:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554422:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554429:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826651.554463:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332767744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554490:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826651.554495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554508:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826651.554549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332767872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826651.554564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554567:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554570:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554574:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826651.554589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332768000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826651.554608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554611:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554615:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554618:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554622:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826651.554633:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332768128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826651.554647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554651:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554654:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826651.554671:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332768256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554679:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554681:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826651.554682:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554685:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554695:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554703:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826651.554718:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332768384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554727:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554730:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826651.554731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554734:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554737:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554741:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554745:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826651.554754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332768512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826651.554762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826651.554765:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826651.554767:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826651.554770:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8462 00000100:00080000:3.0:1787826651.554773:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826651.554776:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826651.554780:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826651.554796:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332768640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826653.986725:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826656.034480:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826658.082436:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826661.154390:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826661.794358:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794366:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826661.794370:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794384:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794394:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794402:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826661.794438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332769792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826661.794467:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794478:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794494:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826661.794507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332769920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794525:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826661.794529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794534:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794538:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794542:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794547:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826661.794559:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332770048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826661.794576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794584:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794588:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794592:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794608:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826661.794623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332770176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794647:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794651:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826661.794653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794660:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794675:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794683:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826661.794705:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332770304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794724:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826661.794726:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794737:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794751:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794759:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826661.794774:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332770432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794805:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794809:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826661.794811:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794819:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794824:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794830:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794839:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826661.794854:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332770560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826661.794865:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826661.794871:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826661.794875:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826661.794882:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8472 00000100:00080000:3.0:1787826661.794888:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826661.794895:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826661.794902:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826661.794914:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332770688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826664.226473:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826666.274480:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826666.914321:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914330:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826666.914333:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914341:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914349:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914355:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826666.914399:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332770816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826666.914440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914444:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914453:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914462:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826666.914475:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332770944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914492:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826666.914494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914501:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914504:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914509:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826666.914518:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332771072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914524:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826666.914527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914529:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914532:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914534:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826666.914542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332771200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914547:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826666.914550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914551:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914553:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914556:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826666.914563:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332771328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914567:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826666.914570:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914572:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914574:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826666.914582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332771456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826666.914590:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914593:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914598:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914607:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826666.914617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332771584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826666.914630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826666.914632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826666.914634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826666.914637:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8477 00000100:00080000:3.0:1787826666.914639:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826666.914643:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826666.914646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826666.914656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332771712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826668.322746:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826671.394837:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826674.466606:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826676.514419:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826677.154515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826677.154538:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154548:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154558:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154565:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826677.154602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332772864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154641:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826677.154643:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154648:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154652:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826677.154677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332772992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154687:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154690:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826677.154692:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154695:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154699:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154702:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154706:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826677.154717:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332773120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154726:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154729:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826677.154731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154734:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154737:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154740:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154745:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826677.154755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332773248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154764:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154766:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826677.154768:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154771:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154774:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154777:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154781:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826677.154791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332773376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154799:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154802:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826677.154803:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154807:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154810:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154814:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154818:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826677.154827:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332773504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154835:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154838:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826677.154839:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154843:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154846:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154850:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154855:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826677.154866:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332773632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826677.154875:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826677.154878:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826677.154880:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826677.154883:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8487 00000100:00080000:3.0:1787826677.154886:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826677.154892:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826677.154901:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826677.154925:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332773760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826678.562418:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826681.634718:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826682.274384:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274393:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826682.274397:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274407:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274418:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274424:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826682.274461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332773888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274490:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826682.274500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274504:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274508:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274512:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274520:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826682.274554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332774016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274568:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826682.274570:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274574:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274581:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274594:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274601:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826682.274614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332774144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826682.274628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274640:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274647:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274653:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826682.274677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332774272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826682.274703:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274710:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274714:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274717:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826682.274735:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332774400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274754:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826682.274762:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274769:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274776:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274782:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274787:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826682.274801:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332774528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274826:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274837:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826682.274842:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274849:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274854:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274859:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274865:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826682.274883:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332774656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826682.274925:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826682.274932:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826682.274935:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826682.274943:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8493 00000100:00080000:0.0:1787826682.274949:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826682.274956:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826682.274962:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826682.274979:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332774784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826684.706425:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826686.754414:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826687.394341:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394349:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826687.394352:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394358:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394367:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394372:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826687.394407:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332774912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394434:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826687.394439:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394443:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394446:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394449:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826687.394466:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332775040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826687.394477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394482:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394487:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826687.394501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332775168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826687.394509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394512:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394515:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394518:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394521:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826687.394528:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332775296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826687.394541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394544:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394547:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394549:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826687.394560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332775424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826687.394574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394578:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394581:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826687.394610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332775552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394627:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826687.394637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394643:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394648:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826687.394675:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332775680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826687.394684:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826687.394688:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826687.394692:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826687.394697:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8498 00000100:00080000:3.0:1787826687.394701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826687.394704:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826687.394710:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826687.394722:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332775808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826688.802456:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826691.874390:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826692.514346:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514354:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826692.514358:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514366:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514375:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826692.514417:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332775936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826692.514449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514456:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514460:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826692.514484:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332776064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826692.514509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514513:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514517:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514522:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826692.514538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332776192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514549:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826692.514554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514558:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514574:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826692.514584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332776320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826692.514621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514624:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514626:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514631:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514637:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826692.514647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332776448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826692.514667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514670:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514674:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826692.514686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332776576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826692.514710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514719:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514728:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514736:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826692.514762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332776704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826692.514772:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826692.514778:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826692.514780:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826692.514787:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8503 00000100:00080000:3.0:1787826692.514795:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826692.514798:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826692.514804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826692.514816:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332776832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826694.946430:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826696.994441:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826697.634429:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826697.634448:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634468:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634481:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634488:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826697.634536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332776960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826697.634572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634576:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634578:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634581:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634591:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826697.634603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332777088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826697.634618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826697.634640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332777216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634649:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826697.634653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634657:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634675:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634680:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826697.634700:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332777344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634710:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634714:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826697.634716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634722:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826697.634746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332777472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634756:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634762:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826697.634767:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634777:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634782:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634794:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634803:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826697.634831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332777600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634842:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634846:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826697.634848:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634852:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634855:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634861:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634868:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826697.634879:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332777728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826697.634887:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826697.634889:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826697.634891:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826697.634895:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8508 00000100:00080000:0.0:1787826697.634899:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826697.634903:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826697.634907:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826697.634917:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332777856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826699.043042:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826702.114409:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826702.754647:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754657:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826702.754661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754670:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754692:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754699:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826702.754735:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332777984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754767:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754780:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826702.754782:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754786:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754790:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754795:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.754804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826702.754818:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332778112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754830:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754833:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826702.754834:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754836:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754839:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754841:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.754845:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826702.754854:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332778240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754860:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754862:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826702.754863:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754866:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754869:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754871:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.754873:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826702.754881:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332778368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754888:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754890:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826702.754892:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754895:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754897:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754899:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.754902:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826702.754908:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332778496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754914:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754916:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826702.754920:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754927:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754932:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.754940:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.754948:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826702.754961:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332778624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.754978:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.754984:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826702.754988:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.754994:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.754998:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.755001:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.755005:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826702.755012:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332778752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826702.755022:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826702.755027:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826702.755028:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826702.755031:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8513 00000100:00080000:0.0:1787826702.755034:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826702.755036:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826702.755039:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826702.755046:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332778880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826705.186406:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826707.234399:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826709.282474:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826712.354409:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826715.426919:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787826716.066502:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787826716.066544:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787826716.066561:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066603:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066611:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066619:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066625:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066631:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066637:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787826716.066643:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787826717.474558:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826719.522431:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826722.594434:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826723.234331:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234340:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826723.234344:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234368:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234378:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234386:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826723.234424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332782080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234457:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826723.234459:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234471:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826723.234496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332782208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826723.234520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234524:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234528:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826723.234546:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332782336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234554:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234557:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826723.234559:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234564:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234567:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234571:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234576:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826723.234588:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332782464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234600:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826723.234603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234606:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234609:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234612:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234616:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826723.234626:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332782592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234646:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234648:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826723.234650:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234653:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234655:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234658:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234661:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826723.234670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332782720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234679:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234687:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826723.234690:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234694:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234710:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234716:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234724:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826723.234738:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332782848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826723.234750:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826723.234755:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826723.234759:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826723.234764:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8533 00000100:00080000:0.0:1787826723.234768:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826723.234772:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826723.234777:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826723.234791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332782976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826725.666527:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826727.714435:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826728.354361:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354372:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826728.354375:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354384:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354394:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354399:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826728.354438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332783104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826728.354468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354476:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354479:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826728.354502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332783232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354519:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826728.354520:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354524:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354529:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354533:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826728.354549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332783360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354558:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826728.354563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354574:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354579:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826728.354608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332783488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354631:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826728.354636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354642:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354646:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354654:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826728.354679:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332783616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826728.354701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354707:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354715:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354722:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354731:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826728.354747:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332783744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354767:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826728.354773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354780:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354786:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354793:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354800:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826728.354826:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332783872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826728.354835:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826728.354838:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826728.354840:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826728.354846:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8539 00000100:00080000:3.0:1787826728.354852:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826728.354855:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826728.354861:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826728.354876:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332784000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826729.762451:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826732.834434:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826733.474600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826733.474617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.474627:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.474639:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.474647:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826733.474687:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332784128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.474712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474716:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826733.474718:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.474723:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.474727:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.474732:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.474744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826733.474760:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332784256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.474775:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474780:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826733.474784:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.474794:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.474798:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.474804:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.474810:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826733.474825:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332784384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.474832:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826733.474837:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.474841:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.474857:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.474861:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.474873:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826733.474893:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332784512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.474904:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474908:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826733.474911:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.474919:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.474925:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.474935:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.474947:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826733.474962:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332784640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.474976:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.474988:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826733.474992:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.475006:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.475017:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.475028:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.475037:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826733.475053:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332784768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.475062:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.475065:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826733.475067:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.475071:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.475073:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.475077:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.475083:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826733.475095:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332784896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826733.475111:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826733.475116:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826733.475118:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826733.475125:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8544 00000100:00080000:3.0:1787826733.475129:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826733.475137:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826733.475147:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826733.475169:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332785024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826735.906602:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826737.954445:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826738.594453:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594462:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826738.594466:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594475:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594485:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594491:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826738.594534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332785152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826738.594572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594580:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594584:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826738.594604:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332785280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826738.594618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826738.594640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332785408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594648:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594651:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826738.594653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594656:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594658:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594662:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826738.594674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332785536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826738.594731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594735:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594738:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594741:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594745:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826738.594756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332785664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594764:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826738.594768:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594772:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594775:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594778:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594782:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826738.594790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332785792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594804:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594807:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826738.594809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594813:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594816:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594819:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594822:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826738.594829:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332785920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826738.594837:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826738.594840:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826738.594841:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826738.594844:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8549 00000100:00080000:3.0:1787826738.594846:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826738.594850:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826738.594855:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826738.594871:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332786048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826740.004920:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826743.075061:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826743.714834:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.714842:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826743.714845:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.714852:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.714860:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.714869:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826743.714900:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332786176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.714924:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.714927:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826743.714928:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.714931:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.714933:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.714936:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.714943:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826743.714951:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332786304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.714957:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.714959:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826743.714959:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.714961:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.714963:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.714964:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.714966:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826743.714971:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332786432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.714976:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.714978:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826743.714979:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.714981:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.714983:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.714985:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.714989:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826743.714999:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332786560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.715006:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.715008:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826743.715010:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.715013:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.715016:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.715020:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.715024:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826743.715033:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332786688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.715042:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.715045:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826743.715047:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.715050:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.715058:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.715062:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.715067:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826743.715077:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332786816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.715085:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.715089:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826743.715090:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.715094:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.715097:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.715101:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.715105:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826743.715115:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332786944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826743.715124:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826743.715127:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826743.715128:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826743.715131:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8554 00000100:00080000:0.0:1787826743.715133:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826743.715136:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826743.715140:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826743.715149:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332787072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826746.146637:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826748.194540:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826748.834337:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834347:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826748.834351:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834376:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826748.834425:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332787200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834475:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834479:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826748.834481:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834485:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834489:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834493:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834501:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826748.834515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332787328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834525:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834528:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826748.834530:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834533:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826748.834554:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332787456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834563:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826748.834567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834570:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834574:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834577:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826748.834591:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332787584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834599:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826748.834604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834608:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834611:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834616:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826748.834634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332787712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834649:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826748.834654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834661:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826748.834680:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332787840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826748.834718:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834723:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834728:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834732:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834737:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826748.834749:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332787968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826748.834759:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826748.834762:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826748.834764:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826748.834768:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8559 00000100:00080000:0.0:1787826748.834772:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826748.834779:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826748.834787:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826748.834802:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332788096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826750.242531:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826753.314560:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826753.954342:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954351:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826753.954355:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954364:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954374:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826753.954424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332788224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954457:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826753.954459:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954466:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954469:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954478:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826753.954488:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332788352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826753.954502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954507:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954513:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954515:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954518:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826753.954526:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332788480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954544:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826753.954550:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954554:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954558:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954565:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954573:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826753.954586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332788608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954600:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826753.954604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954614:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954619:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954626:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954634:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826753.954647:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332788736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954654:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826753.954658:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954660:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954663:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954667:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954671:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826753.954690:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332788864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954706:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826753.954712:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954720:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954726:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954734:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954738:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826753.954751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332788992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826753.954768:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826753.954773:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826753.954777:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826753.954781:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8564 00000100:00080000:0.0:1787826753.954783:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826753.954786:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826753.954789:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826753.954798:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332789120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826756.386437:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826758.434439:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826759.074362:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074372:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826759.074376:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074404:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074414:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074420:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826759.074456:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332789248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074486:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074490:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826759.074491:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074495:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074498:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074502:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074511:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826759.074528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332789376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074545:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826759.074547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074551:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074555:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074559:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074565:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826759.074577:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332789504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826759.074598:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074602:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074606:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074610:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826759.074629:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332789632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074643:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074653:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826759.074657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074665:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074673:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074680:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074687:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826759.074698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332789760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074716:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074719:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826759.074721:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074726:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074731:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074737:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074744:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826759.074760:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332789888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074800:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826759.074802:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074809:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074814:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074818:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074825:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826759.074842:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332790016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826759.074871:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826759.074878:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826759.074883:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826759.074890:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8569 00000100:00080000:0.0:1787826759.074896:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826759.074903:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826759.074909:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826759.074921:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332790144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826760.482475:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826763.554472:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826766.626496:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826768.674447:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826770.722494:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826773.794449:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826774.434366:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434376:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826774.434380:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434391:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434404:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434413:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826774.434464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332792320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434487:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826774.434488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434492:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434495:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434498:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826774.434516:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332792448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434524:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826774.434530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434536:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434538:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826774.434552:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332792576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826774.434561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434564:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434566:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434568:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826774.434591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332792704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826774.434607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434614:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434624:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826774.434640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332792832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434653:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826774.434656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434662:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434668:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826774.434695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332792960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434715:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434721:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826774.434723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434738:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434744:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434750:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826774.434762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332793088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826774.434775:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826774.434780:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826774.434782:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826774.434787:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8585 00000100:00080000:3.0:1787826774.434793:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826774.434799:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826774.434807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826774.434822:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332793216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826776.866451:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826778.914455:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826779.554363:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554371:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826779.554375:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554384:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554394:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554401:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826779.554439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332793344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826779.554470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554478:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554481:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554490:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826779.554501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332793472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826779.554514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554520:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554527:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826779.554540:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332793600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554548:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826779.554573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554580:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826779.554604:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332793728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826779.554618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554621:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554624:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554628:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826779.554643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332793856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554661:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826779.554663:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554667:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554670:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554675:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826779.554688:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332793984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826779.554701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554704:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554707:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554710:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826779.554724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332794112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826779.554741:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826779.554745:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826779.554747:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826779.554751:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8590 00000100:00080000:3.0:1787826779.554755:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826779.554758:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826779.554763:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826779.554772:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332794240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826780.962424:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787826782.626566:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787826782.626607:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787826782.626627:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626679:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626687:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626695:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626703:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626711:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626718:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826782.626725:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:3.0:1787826784.035516:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826787.106551:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826789.154487:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826791.202447:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826794.275191:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826794.914343:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914353:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826794.914357:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914368:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914378:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914385:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826794.914420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332796416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914455:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826794.914457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914469:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914484:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826794.914499:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332796544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914516:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826794.914518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914521:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914525:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914529:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914540:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826794.914553:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332796672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914567:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914571:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826794.914573:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914578:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914582:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914586:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914601:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826794.914618:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332796800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914646:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826794.914651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914658:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914664:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914671:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914683:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826794.914695:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332796928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914714:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914718:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826794.914720:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914725:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914732:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914737:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914742:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826794.914757:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332797056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914788:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914794:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826794.914797:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914804:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914809:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914818:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914829:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826794.914845:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332797184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826794.914881:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826794.914888:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826794.914893:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826794.914901:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8605 00000100:00080000:3.0:1787826794.914905:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826794.914911:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826794.914917:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826794.914931:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332797312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826797.346491:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826799.394452:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826801.442480:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826804.514345:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826807.586404:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826809.634444:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826811.682429:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826814.754418:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826815.394396:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394405:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826815.394408:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394418:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394428:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394435:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826815.394471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332800512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394503:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394506:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826815.394508:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394512:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394515:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394518:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394528:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826815.394540:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332800640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394552:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394557:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826815.394560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394563:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394567:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394570:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394576:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826815.394587:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332800768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394598:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826815.394600:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394603:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394607:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394611:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394619:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826815.394632:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332800896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394654:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826815.394658:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394662:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394666:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394673:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394681:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826815.394695:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332801024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394710:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394713:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826815.394715:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394726:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394730:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394739:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394750:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826815.394762:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332801152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394778:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394782:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826815.394784:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394789:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394793:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394799:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394808:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826815.394822:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332801280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826815.394844:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826815.394849:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826815.394853:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826815.394861:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8626 00000100:00080000:0.0:1787826815.394866:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826815.394874:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826815.394883:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826815.394897:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332801408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826817.826637:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826819.874470:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826821.922458:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826824.994588:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826828.067384:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826830.114437:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826830.754327:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826830.754341:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754360:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754367:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826830.754405:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332803584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826830.754445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754449:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754452:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754457:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754465:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826830.754478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332803712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826830.754491:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754495:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754501:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754505:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826830.754514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332803840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826830.754527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754530:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754533:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754536:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826830.754550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332803968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826830.754563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826830.754585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332804096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754594:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754597:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826830.754598:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754602:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754609:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826830.754622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332804224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826830.754635:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754639:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754641:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754644:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826830.754658:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332804352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826830.754667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826830.754669:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826830.754671:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826830.754674:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8641 00000100:00080000:3.0:1787826830.754677:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826830.754680:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826830.754684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826830.754694:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332804480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826832.162551:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826835.234402:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826835.874320:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874330:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826835.874345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874355:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874364:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874371:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826835.874418:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332804608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874445:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874448:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826835.874450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874457:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826835.874485:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332804736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826835.874504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874507:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874511:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874518:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826835.874539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332804864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826835.874558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874568:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826835.874586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332804992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826835.874612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874619:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874636:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826835.874653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332805120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874668:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826835.874672:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874678:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874683:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874692:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826835.874723:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332805248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874770:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826835.874774:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874781:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874788:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874796:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826835.874821:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332805376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826835.874831:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826835.874834:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826835.874835:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826835.874843:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8646 00000100:00080000:3.0:1787826835.874860:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826835.874868:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826835.874877:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826835.874893:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332805504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826838.306488:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826840.354628:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826840.994382:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994394:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826840.994398:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994407:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994415:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994421:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826840.994459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332805632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826840.994489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994493:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994501:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994511:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826840.994539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332805760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994554:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826840.994556:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994559:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994563:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994567:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826840.994586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332805888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826840.994606:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994610:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994613:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994616:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826840.994631:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332806016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826840.994675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994685:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994691:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994699:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826840.994723:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332806144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994741:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994745:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826840.994747:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994753:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994759:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994766:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994771:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826840.994784:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332806272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994831:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994834:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826840.994835:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994839:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994842:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994845:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994849:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826840.994859:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332806400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826840.994875:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826840.994880:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826840.994885:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826840.994891:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8651 00000100:00080000:0.0:1787826840.994896:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826840.994900:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826840.994905:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826840.994916:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332806528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826842.402933:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826845.474418:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826846.114375:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114384:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826846.114389:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114398:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114406:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114412:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826846.114448:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332806656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114473:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826846.114478:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114482:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114486:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114489:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114498:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826846.114510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332806784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826846.114539:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114543:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114547:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114551:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114556:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826846.114569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332806912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114587:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826846.114590:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114593:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114600:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826846.114615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332807040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826846.114628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114633:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114636:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114640:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826846.114661:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332807168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826846.114677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114682:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114687:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114695:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826846.114714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332807296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114732:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826846.114736:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114740:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114745:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114749:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114754:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826846.114770:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332807424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826846.114797:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826846.114800:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826846.114802:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826846.114806:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8656 00000100:00080000:3.0:1787826846.114812:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826846.114820:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826846.114826:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826846.114838:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332807552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826848.546510:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826850.594426:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826851.234368:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234376:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826851.234379:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234386:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234395:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234400:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826851.234428:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332807680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234452:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826851.234454:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234456:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234459:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234471:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826851.234483:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332807808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234502:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234506:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826851.234507:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234511:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234514:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234517:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234522:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826851.234536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332807936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234544:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826851.234545:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234552:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234558:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826851.234568:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332808064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826851.234578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234581:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234585:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234591:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826851.234605:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332808192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234638:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826851.234639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234651:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234659:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234662:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826851.234671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332808320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826851.234687:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234690:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234693:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234698:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234703:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826851.234713:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332808448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826851.234733:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826851.234736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826851.234737:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826851.234752:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8661 00000100:00080000:0.0:1787826851.234758:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826851.234765:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826851.234774:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826851.234792:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332808576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826852.642414:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826855.714445:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826856.354378:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354387:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826856.354390:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354399:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354409:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826856.354451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332808704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826856.354478:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354481:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354484:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354486:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826856.354501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332808832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354509:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826856.354510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354512:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354516:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354519:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826856.354532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332808960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354541:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826856.354544:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354547:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354549:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354553:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826856.354571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332809088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354591:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826856.354593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354597:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354601:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354605:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354609:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826856.354618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332809216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826856.354636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354640:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354643:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354646:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826856.354656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332809344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354661:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826856.354667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354673:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354676:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354681:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354687:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826856.354695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332809472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826856.354716:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826856.354722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826856.354725:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826856.354731:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8667 00000100:00080000:3.0:1787826856.354735:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826856.354739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826856.354744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826856.354756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332809600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826858.786438:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826860.834423:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826861.474350:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474360:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826861.474364:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474373:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474390:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826861.474427:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332809728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474453:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826861.474458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474463:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474480:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826861.474492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332809856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826861.474508:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474513:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474515:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474518:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826861.474527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332809984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474536:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826861.474541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474544:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474550:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474554:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826861.474572:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332810112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826861.474585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474589:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474592:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474595:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826861.474608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332810240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826861.474623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474626:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474634:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826861.474647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332810368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826861.474679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474688:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474691:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474694:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474698:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826861.474708:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332810496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826861.474720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826861.474722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826861.474724:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826861.474727:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8672 00000100:00080000:3.0:1787826861.474731:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826861.474738:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826861.474744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826861.474759:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332810624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826862.882833:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826865.954505:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826866.594343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.594354:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826866.594357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.594366:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.594376:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.594383:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826866.594420:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332810752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.594581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.594585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826866.594587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.594592:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.594595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.594600:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.594608:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826866.594621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332810880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598376:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598382:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826866.598384:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598389:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598395:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598402:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598410:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826866.598432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332811008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598445:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598448:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826866.598450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598453:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598457:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598460:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598465:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826866.598474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332811136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598485:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826866.598487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598491:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598494:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598497:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826866.598511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332811264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598519:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598522:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826866.598523:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598527:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598530:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598534:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598538:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826866.598546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332811392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826866.598559:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598562:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598565:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598569:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598573:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826866.598582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332811520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826866.598596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826866.598599:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826866.598600:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826866.598604:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8677 00000100:00080000:3.0:1787826866.598606:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826866.598609:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826866.598613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826866.598622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332811648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826869.026477:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826871.074497:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826873.122473:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826876.194412:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826879.266480:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826881.314437:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826883.362415:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826886.434419:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826887.074363:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074372:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826887.074376:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074385:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074395:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074401:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826887.074436:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332814848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074469:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074471:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826887.074473:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074476:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074479:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074481:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074488:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826887.074510:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332814976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826887.074524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074530:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074534:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074541:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826887.074552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332815104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074561:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074564:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826887.074566:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074570:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074582:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074586:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074592:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826887.074606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332815232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074615:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074618:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826887.074620:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074624:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074628:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074634:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074642:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826887.074654:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332815360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074678:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074684:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826887.074686:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074696:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074703:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074722:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826887.074743:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332815488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074750:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826887.074754:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074758:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074763:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074766:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074770:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826887.074778:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332815616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826887.074809:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826887.074813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826887.074817:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826887.074825:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8697 00000100:00080000:3.0:1787826887.074831:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826887.074847:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826887.074855:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826887.074870:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332815744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826889.506571:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826891.554443:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826892.194335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826892.194346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194353:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194362:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194369:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826892.194397:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332815872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194416:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194419:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826892.194420:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194423:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194424:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194427:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194433:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826892.194443:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332816000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826892.194452:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194467:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194470:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194477:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826892.194499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332816128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826892.194514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194518:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194521:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194526:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194530:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826892.194543:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332816256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826892.194554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194556:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194559:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826892.194570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332816384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194577:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826892.194579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194582:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194584:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826892.194602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332816512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194613:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826892.194614:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826892.194642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332816640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826892.194657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826892.194659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826892.194662:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826892.194668:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8702 00000100:00080000:0.0:1787826892.194671:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826892.194673:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826892.194675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826892.194681:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332816768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826893.602467:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826896.674900:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826897.314341:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314349:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826897.314352:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314361:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314389:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826897.314436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332816896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826897.314475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314480:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826897.314511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332817024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826897.314526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314529:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314532:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314535:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314538:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826897.314545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332817152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826897.314555:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314557:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314558:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314560:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826897.314568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332817280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826897.314580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314583:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314586:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314590:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314593:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826897.314603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332817408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826897.314616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314626:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826897.314647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332817536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314657:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826897.314658:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314661:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314663:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314666:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314669:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826897.314678:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332817664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826897.314686:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826897.314688:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826897.314690:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826897.314693:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8708 00000100:00080000:0.0:1787826897.314696:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826897.314700:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826897.314703:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826897.314713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332817792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787826899.747211:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826901.794435:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826902.434332:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826902.434342:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434357:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434362:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826902.434406:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332817920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434429:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434431:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826902.434433:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434435:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434438:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434440:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434446:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826902.434455:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332818048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826902.434465:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434467:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434469:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434474:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826902.434481:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332818176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826902.434493:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434496:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434498:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434500:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434503:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826902.434509:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332818304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434516:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826902.434517:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434519:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434521:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434523:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826902.434532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332818432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434536:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826902.434540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434544:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434546:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826902.434566:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332818560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434579:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826902.434581:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434586:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434589:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434593:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826902.434618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332818688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826902.434643:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826902.434647:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826902.434651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826902.434660:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8713 00000100:00080000:0.0:1787826902.434666:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826902.434674:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826902.434682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826902.434703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332818816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826903.842517:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826906.914503:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826907.554359:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554370:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826907.554374:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554395:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554405:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554413:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826907.554445:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332818944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554465:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826907.554469:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554472:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554474:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554476:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826907.554490:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332819072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826907.554501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554503:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554505:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554507:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554510:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826907.554520:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332819200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826907.554533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554536:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554538:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554539:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826907.554549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332819328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554554:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554556:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826907.554557:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554559:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554573:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554575:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554578:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826907.554586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332819456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826907.554597:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554600:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554605:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554607:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826907.554617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332819584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554623:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554626:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826907.554630:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554636:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554640:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554644:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826907.554656:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332819712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826907.554665:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826907.554668:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826907.554670:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826907.554676:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8718 00000100:00080000:3.0:1787826907.554681:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826907.554691:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826907.554700:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826907.554717:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332819840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826909.986529:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826912.034482:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826912.674357:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674365:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826912.674369:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674378:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674388:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674403:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826912.674439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332819968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674470:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826912.674472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674479:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674498:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826912.674511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332820096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826912.674534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674539:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674543:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674547:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826912.674562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332820224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674572:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674576:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826912.674578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674581:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674585:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674589:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826912.674616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332820352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674635:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826912.674639:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674644:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674650:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674654:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826912.674685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332820480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826912.674705:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674710:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674715:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674720:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826912.674755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332820608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674767:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826912.674773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674781:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674787:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674792:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826912.674813:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332820736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826912.674837:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826912.674841:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826912.674843:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826912.674846:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8723 00000100:00080000:3.0:1787826912.674850:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826912.674854:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826912.674859:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826912.674871:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332820864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826914.082493:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787826915.746583:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787826915.746643:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787826915.746668:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746721:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746732:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746741:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746762:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746772:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746780:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826915.746789:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:3.0:1787826917.154368:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826917.794397:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826917.794411:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794422:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794431:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794437:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826917.794476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332820992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794509:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826917.794511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794515:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794519:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794523:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826917.794544:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332821120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794555:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826917.794564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794571:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794574:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794578:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794583:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826917.794597:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332821248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794606:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794609:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826917.794616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794621:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794631:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794637:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826917.794656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332821376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826917.794678:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794691:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794697:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826917.794722:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332821504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794740:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794743:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826917.794745:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794756:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794763:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794769:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794775:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826917.794787:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332821632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794796:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794800:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826917.794803:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794809:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794813:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794817:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794824:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826917.794842:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332821760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826917.794866:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826917.794875:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826917.794878:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826917.794884:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8728 00000100:00080000:0.0:1787826917.794891:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826917.794899:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826917.794906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826917.794922:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332821888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826920.226429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826922.274443:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826922.914386:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914398:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826922.914402:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914415:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914428:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914437:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826922.914479:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332822016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914510:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914513:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826922.914515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914519:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914522:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914526:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914538:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826922.914550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332822144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914563:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826922.914566:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914570:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914573:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914577:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826922.914591:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332822272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914599:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826922.914603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914606:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914608:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914610:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914613:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826922.914620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332822400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914629:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826922.914631:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914633:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914643:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914645:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914648:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826922.914657:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332822528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914665:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914669:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826922.914671:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914675:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914678:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914681:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914684:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826922.914690:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332822656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914695:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914698:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826922.914700:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914702:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914705:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914709:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914714:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826922.914729:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332822784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826922.914767:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826922.914782:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826922.914793:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826922.914809:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8733 00000100:00080000:3.0:1787826922.914815:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826922.914820:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826922.914827:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826922.914846:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332822912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826924.322405:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826927.394543:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826928.034346:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034357:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826928.034361:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034370:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034380:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034391:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826928.034433:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332823040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034461:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034464:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826928.034466:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034470:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034473:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034477:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826928.034498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332823168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034509:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034515:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826928.034519:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034523:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034526:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034530:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034535:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826928.034547:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332823296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034555:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826928.034560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034563:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034567:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034572:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034578:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826928.034589:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332823424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034600:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826928.034602:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034607:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034611:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034618:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034625:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826928.034638:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332823552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034662:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826928.034667:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034674:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034679:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034683:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034687:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826928.034700:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332823680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826928.034716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034719:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034731:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034736:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034742:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826928.034755:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332823808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826928.034767:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826928.034774:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826928.034777:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826928.034794:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8738 00000100:00080000:3.0:1787826928.034800:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826928.034807:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826928.034813:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826928.034828:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332823936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826930.466460:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826932.514446:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826933.154361:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154374:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826933.154379:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154391:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154403:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154412:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826933.154456:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332824064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154492:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826933.154494:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154498:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154502:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154506:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154518:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826933.154529:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332824192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154538:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154540:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826933.154542:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154545:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154548:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154552:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154556:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826933.154568:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332824320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154575:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154578:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826933.154579:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154582:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154585:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154588:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154593:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826933.154600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332824448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154608:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154611:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826933.154612:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154616:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154619:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154622:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826933.154634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332824576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826933.154657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154662:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154665:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154668:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154673:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826933.154682:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332824704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154691:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154694:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826933.154696:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154702:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154706:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154710:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826933.154718:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332824832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826933.154734:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826933.154739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826933.154743:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826933.154750:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8743 00000100:00080000:3.0:1787826933.154758:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826933.154765:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826933.154769:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826933.154782:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332824960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826934.562373:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826937.634405:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826938.274356:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274363:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826938.274366:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274374:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274383:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274388:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826938.274422:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332825088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826938.274454:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274457:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274460:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274471:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826938.274481:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332825216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274491:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826938.274492:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274496:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274499:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274503:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274508:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826938.274514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332825344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274519:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826938.274523:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274525:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274526:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274528:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826938.274537:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332825472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274543:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826938.274549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274552:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274555:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274559:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274564:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826938.274578:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332825600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274590:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826938.274592:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274596:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274599:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274603:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274608:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826938.274622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332825728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274635:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274640:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826938.274644:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274652:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274656:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274664:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274672:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826938.274688:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332825856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826938.274700:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826938.274705:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826938.274709:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826938.274714:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8749 00000100:00080000:0.0:1787826938.274716:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826938.274719:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826938.274723:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826938.274731:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332825984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826940.706428:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826942.754500:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826943.394822:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.394832:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826943.394835:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.394852:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.394862:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.394868:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826943.394905:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332826112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.394933:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.394936:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826943.394938:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395005:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395010:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395015:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395024:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826943.395038:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332826240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395055:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395059:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826943.395061:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395070:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395074:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395078:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395084:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826943.395109:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332826368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395128:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395132:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826943.395134:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395142:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395148:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395153:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395161:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826943.395175:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332826496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395182:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395184:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826943.395185:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395188:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395190:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395195:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395203:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826943.395216:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332826624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395270:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395275:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826943.395277:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395284:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395291:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395297:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395304:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826943.395317:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332826752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826943.395345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395350:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395783:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395789:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826943.395808:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332826880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826943.395818:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826943.395821:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826943.395823:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826943.395826:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8754 00000100:00080000:3.0:1787826943.395829:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826943.395833:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826943.395837:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826943.395847:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332827008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826944.802475:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826947.874408:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826948.514335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514344:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826948.514347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514356:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826948.514410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332827136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514443:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826948.514449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514456:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826948.514480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332827264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514494:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826948.514496:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514503:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826948.514525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332827392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826948.514574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514587:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514591:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826948.514613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332827520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826948.514628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514632:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514637:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514644:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826948.514681:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332827648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514701:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514707:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826948.514709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514717:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514721:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514725:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826948.514743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332827776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514760:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826948.514763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514769:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514775:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514783:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514792:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826948.514807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332827904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826948.514836:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826948.514839:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826948.514841:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826948.514845:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8759 00000100:00080000:3.0:1787826948.514849:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826948.514856:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826948.514862:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826948.514873:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332828032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826950.946499:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826952.994395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826953.634520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826953.634533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634543:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634553:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634559:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826953.634591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332828160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826953.634618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634625:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826953.634640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332828288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634647:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826953.634650:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634654:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634659:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826953.634674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332828416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826953.634696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634700:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634703:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634707:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826953.634733:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332828544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634747:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634751:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826953.634753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634758:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634764:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634768:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634773:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826953.634800:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332828672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634814:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634820:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826953.634824:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634832:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634837:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634842:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634848:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826953.634861:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332828800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634891:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634899:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826953.634902:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634913:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634919:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634927:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634934:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826953.634952:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332828928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826953.634964:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826953.634971:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826953.634973:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826953.634977:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8764 00000100:00080000:3.0:1787826953.634989:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826953.634993:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826953.634999:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826953.635012:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332829056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826955.042451:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826958.114413:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826958.754367:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754377:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826958.754380:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754391:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754400:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754406:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826958.754439:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332829184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754464:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754467:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826958.754469:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754473:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754476:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754480:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754488:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826958.754501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332829312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826958.754521:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754528:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826958.754548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332829440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754557:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826958.754562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754566:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754571:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754574:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826958.754594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332829568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826958.754608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754612:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754615:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754619:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754624:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826958.754636:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332829696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754658:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754663:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826958.754664:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754671:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754677:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754684:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754689:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826958.754702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332829824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754723:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754726:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826958.754728:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754746:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754751:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754757:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754761:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826958.754773:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332829952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826958.754782:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826958.754785:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826958.754786:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826958.754789:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8769 00000100:00080000:0.0:1787826958.754792:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826958.754795:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826958.754798:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826958.754807:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332830080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826961.186719:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826963.234776:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826963.874347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874358:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826963.874362:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874382:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874389:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826963.874424:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332830208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826963.874456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826963.874491:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332830336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826963.874510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874521:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874524:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826963.874542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332830464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826963.874557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874571:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826963.874591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332830592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874607:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826963.874612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874625:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826963.874647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332830720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874664:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826963.874668:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874676:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874682:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874688:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874695:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826963.874710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332830848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826963.874725:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874737:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874741:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826963.874756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332830976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826963.874768:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826963.874771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826963.874773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826963.874779:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8774 00000100:00080000:3.0:1787826963.874784:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826963.874789:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826963.874794:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826963.874807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332831104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826965.282403:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826968.354493:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826968.994339:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994348:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826968.994352:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994382:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994390:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826968.994428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332831232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994457:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994462:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826968.994464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994468:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994484:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826968.994511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332831360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826968.994525:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994529:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994533:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994536:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826968.994552:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332831488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826968.994566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994570:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994574:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994578:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994583:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826968.994596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332831616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994613:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826968.994615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994619:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994623:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994631:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826968.994642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332831744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826968.994657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994678:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994683:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994687:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994693:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826968.994706:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332831872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826968.994726:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994735:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994741:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994748:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994756:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826968.994771:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332832000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826968.994785:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826968.994792:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826968.994794:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826968.994799:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8779 00000100:00080000:3.0:1787826968.994803:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826968.994806:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826968.994812:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826968.994829:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332832128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826971.426412:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826973.474382:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826974.114333:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826974.114345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114364:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114375:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114382:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826974.114419:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332832256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114460:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826974.114462:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114471:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114475:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826974.114495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332832384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114503:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826974.114509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114512:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114515:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114518:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114522:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826974.114533:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332832512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826974.114561:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114568:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114571:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114576:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826974.114587:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332832640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114598:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826974.114600:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114603:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114606:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114610:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826974.114623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332832768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826974.114635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114639:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114642:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826974.114658:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332832896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114672:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114689:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826974.114690:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114694:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114697:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114700:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114705:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826974.114715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332833024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826974.114724:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826974.114727:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826974.114728:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826974.114731:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8784 00000100:00080000:3.0:1787826974.114734:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826974.114738:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826974.114742:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826974.114751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332833152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826975.522500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826978.597950:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826979.234358:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234372:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826979.234376:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234388:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234402:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234411:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826979.234462:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332833280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234494:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826979.234500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234505:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234510:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826979.234541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332833408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826979.234564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234568:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826979.234595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332833536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234603:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826979.234608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234611:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234624:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826979.234648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332833664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234658:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234661:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826979.234662:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234666:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234669:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234672:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234677:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826979.234691:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332833792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234705:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234709:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826979.234712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234717:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234721:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234728:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234738:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826979.234753:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332833920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234764:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234773:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826979.234775:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234780:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234784:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234789:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234794:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826979.234809:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332834048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826979.234822:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826979.234826:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826979.234828:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826979.234834:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8789 00000100:00080000:3.0:1787826979.234839:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826979.234844:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826979.234863:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826979.234877:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332834176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826981.666436:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787826982.306600:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787826982.306655:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787826982.306672:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306716:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306724:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306732:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306738:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306746:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306753:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787826982.306760:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787826983.714455:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787826984.354359:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354369:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787826984.354373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354382:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354392:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354399:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787826984.354438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332834304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826984.354610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354614:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354618:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826984.354651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332834432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354660:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826984.354665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354668:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354671:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354674:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826984.354689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332834560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354703:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787826984.354709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354711:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354714:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354717:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354721:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826984.354730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332834688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354755:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826984.354756:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354759:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354762:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354766:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354770:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826984.354779:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332834816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354787:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354790:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787826984.354792:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354796:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354799:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354802:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354828:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826984.354838:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332834944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354846:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354850:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787826984.354851:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354855:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354858:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354861:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354864:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787826984.354873:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332835072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787826984.354882:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787826984.354884:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787826984.354886:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787826984.354889:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8795 00000100:00080000:3.0:1787826984.354892:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787826984.354906:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787826984.354910:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787826984.354920:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332835200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787826985.762457:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826988.834453:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826991.906502:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826993.954457:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787826996.002601:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787826999.074424:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787826999.714406:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714421:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787826999.714425:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714439:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714452:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714461:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787826999.714522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f480 x1874663332837376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714564:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714568:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826999.714571:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714576:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714580:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714584:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826999.714612:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332837504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714623:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714626:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826999.714627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714631:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714634:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714637:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714642:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826999.714653:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332837632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714665:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787826999.714667:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714671:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714675:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714679:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714684:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826999.714696:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332837760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714719:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826999.714720:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714725:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714728:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714732:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714737:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826999.714762:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332837888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714772:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714775:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787826999.714777:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714781:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714785:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714789:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714794:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826999.714804:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332838016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714813:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714816:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787826999.714818:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714822:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714825:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714829:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714834:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787826999.714844:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332838144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787826999.714854:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787826999.714857:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787826999.714859:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787826999.714863:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8810 00000100:00080000:0.0:1787826999.714866:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787826999.714869:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787826999.714874:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787826999.714884:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332838272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827002.146557:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827004.194574:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827004.834440:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827004.834459:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834472:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834485:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834494:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827004.834541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332838400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834584:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827004.834585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834594:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834598:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834609:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827004.834622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332838528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827004.834636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834639:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834642:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834651:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827004.834663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332838656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834684:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834688:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827004.834690:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834697:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834702:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834706:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834711:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827004.834723:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332838784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834732:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834735:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827004.834737:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834740:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834744:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834747:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834752:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827004.834762:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332838912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834768:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834770:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827004.834771:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834773:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834777:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834781:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834786:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827004.834796:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332839040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834810:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827004.834814:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834818:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834821:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834824:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834828:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827004.834838:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332839168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827004.834847:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827004.834850:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827004.834852:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827004.834855:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8815 00000100:00080000:0.0:1787827004.834857:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827004.834861:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827004.834864:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827004.834872:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332839296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827006.242461:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827009.314412:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827012.386457:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827014.434459:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827016.482446:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827020.194340:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194351:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827020.194354:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194364:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194374:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194389:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827020.194424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332841472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827020.194457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194469:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194478:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827020.194498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332841600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827020.194520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194523:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194527:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194531:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827020.194548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332841728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194562:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827020.194578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194586:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194590:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194594:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827020.194616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332841856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194648:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827020.194653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194660:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194666:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194673:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194680:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827020.194695:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332841984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194707:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827020.194712:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194718:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194726:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194732:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194737:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827020.194764:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332842112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194793:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194799:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827020.194803:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194812:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194818:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194827:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194836:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827020.194863:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332842240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827020.194879:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827020.194883:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827020.194884:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827020.194888:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8830 00000100:00080000:0.0:1787827020.194892:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827020.194903:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827020.194911:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827020.194925:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332842368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827022.626433:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827024.674441:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827026.722447:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827030.434357:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434367:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827030.434371:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434381:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434396:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434404:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827030.434435:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332843520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434455:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434458:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827030.434460:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434479:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827030.434492:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332843648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827030.434523:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434529:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434533:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434543:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827030.434562:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332843776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434579:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827030.434581:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434585:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434589:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434592:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827030.434604:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332843904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434610:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434612:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827030.434613:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434615:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434617:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434619:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434622:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827030.434629:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332844032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827030.434658:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434662:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434669:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434674:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434680:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827030.434696:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332844160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434706:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827030.434716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434720:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434724:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434727:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434732:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827030.434751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332844288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827030.434761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827030.434766:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827030.434771:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827030.434778:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8841 00000100:00080000:0.0:1787827030.434782:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827030.434789:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827030.434797:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827030.434812:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332844416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827032.866566:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827034.914436:0:230433:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1787827034.914477:0:230433:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1787827034.914505:0:49150:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787827034.914526:0:49150:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801def lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x526:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0xe21309aa81f4ce11 expref: -99 pid: 230036 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827034.914558:0:49150:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x6eb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827034.914571:0:49150:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x6eb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827034.914632:0:49150:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801def lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x526:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xe21309aa81f4ce11 expref: -99 pid: 230036 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827034.914646:0:49150:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827034.914653:0:49150:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9057c7539000 lock: 00000000b29ec442/0x95194e8760801def lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x526:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xe21309aa81f4ce11 expref: -99 pid: 230036 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827034.914679:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827034.914683:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827034.914686:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827041.058451:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827045.154411:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827048.866580:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827048.866625:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827048.866640:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866674:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866757:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866770:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866784:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866796:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866819:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827048.866836:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00080000:3.0:1787827050.914377:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914387:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827050.914390:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914400:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914409:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827050.914460:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332847872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914496:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827050.914498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914503:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914508:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914513:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827050.914539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332848000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827050.914565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914568:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827050.914597:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332848128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914607:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827050.914612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914617:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914623:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914637:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827050.914650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332848256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914684:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914689:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827050.914693:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914708:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914716:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827050.914746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332848384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914766:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914774:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827050.914781:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914787:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914794:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914802:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914808:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827050.914824:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332848512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914838:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914846:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827050.914850:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914857:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914863:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914871:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914879:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827050.914889:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332848640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827050.914906:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827050.914910:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827050.914912:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827050.914920:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8861 00000100:00080000:3.0:1787827050.914925:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827050.914931:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827050.914938:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827050.914955:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332848768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827051.298431:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827055.394441:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827056.034359:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827056.034372:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034380:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034390:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034396:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827056.034429:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332848896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034460:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827056.034462:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034466:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034469:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034472:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034481:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827056.034492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332849024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034511:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827056.034516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034523:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034527:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827056.034541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332849152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827056.034555:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034558:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827056.034575:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332849280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827056.034583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034585:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034586:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034591:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827056.034603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332849408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827056.034617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034622:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827056.034654:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332849536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034665:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827056.034679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034685:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034691:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034694:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034699:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827056.034713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332849664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827056.034729:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827056.034732:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827056.034734:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827056.034738:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8866 00000100:00080000:3.0:1787827056.034742:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827056.034749:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827056.034756:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827056.034773:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332849792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154334:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154345:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827061.154348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154358:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154368:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827061.154410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332849920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154448:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827061.154450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154454:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154457:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827061.154486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332850048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154511:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154515:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827061.154517:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154531:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154535:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827061.154555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332850176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154564:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827061.154569:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154573:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154577:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154583:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154590:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827061.154599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332850304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827061.154619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154624:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827061.154662:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332850432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154671:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827061.154688:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154692:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154695:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154699:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154703:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827061.154715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332850560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154743:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154750:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827061.154753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154759:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154764:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154771:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154778:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827061.154791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332850688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827061.154807:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827061.154815:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827061.154818:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827061.154823:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8871 00000100:00080000:3.0:1787827061.154826:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827061.154829:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827061.154833:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827061.154842:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332850816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827061.538765:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827065.634481:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827071.778392:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827075.874462:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827082.018450:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827086.114436:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827086.754368:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754379:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827086.754384:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754405:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754419:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754428:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827086.754483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332855040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827086.754527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754533:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754537:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754541:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754554:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827086.754569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332855168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754579:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827086.754583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754585:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754589:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754592:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827086.754607:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332855296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754625:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827086.754634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754638:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827086.754652:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332855424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827086.754661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754665:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754669:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754672:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827086.754686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332855552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827086.754704:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754718:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754723:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754728:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754733:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827086.754746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332855680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754756:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754760:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827086.754762:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754771:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754778:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754790:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754802:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827086.754820:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332855808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827086.754833:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827086.754840:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827086.754846:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827086.754857:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8897 00000100:00080000:3.0:1787827086.754865:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827086.754877:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827086.754886:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827086.754905:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332855936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874368:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874382:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827091.874387:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874412:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874426:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874436:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827091.874496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332856064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827091.874537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874546:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874550:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827091.874577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332856192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827091.874595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874598:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874602:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874611:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827091.874642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332856320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874657:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827091.874660:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874664:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874680:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874686:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874693:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827091.874715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332856448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874728:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874732:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827091.874734:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874739:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874744:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874748:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874753:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827091.874769:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332856576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874822:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874828:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827091.874830:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874838:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874845:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874857:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874867:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827091.874885:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332856704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874905:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874911:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827091.874913:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874919:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874922:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874927:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.874932:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827091.874952:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332856832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827091.874964:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827091.874968:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827091.874969:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827091.874979:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8902 00000100:00080000:0.0:1787827091.874988:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827091.874993:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827091.875000:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827091.875014:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332856960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827092.258418:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827096.354436:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827096.994314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827096.994331:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994342:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994352:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994360:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827096.994396:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332857088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994422:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994425:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827096.994426:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994430:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994436:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827096.994456:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332857216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827096.994472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994486:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994491:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827096.994502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332857344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827096.994520:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994528:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994531:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827096.994550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332857472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827096.994564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994568:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994571:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994575:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827096.994592:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332857600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827096.994617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994637:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994644:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827096.994658:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332857728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994678:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827096.994680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994685:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994689:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994697:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827096.994718:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332857856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827096.994737:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827096.994741:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827096.994745:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827096.994753:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8907 00000100:00080000:3.0:1787827096.994757:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827096.994761:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827096.994765:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827096.994775:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332857984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827102.498396:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827106.594407:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827107.234379:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234389:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827107.234393:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234403:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234415:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234429:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827107.234465:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332859136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827107.234501:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234505:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234509:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234513:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234521:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827107.234535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332859264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234549:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234554:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827107.234556:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234560:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234564:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827107.234592:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332859392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827107.234609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234624:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827107.234640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332859520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827107.234663:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234669:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234675:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234683:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234689:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827107.234720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332859648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234732:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234736:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827107.234738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234742:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234749:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234755:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234761:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827107.234776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332859776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234806:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234811:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827107.234815:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234824:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234828:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234831:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234837:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827107.234850:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332859904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827107.234883:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827107.234891:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827107.234897:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827107.234905:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8917 00000100:00080000:0.0:1787827107.234910:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827107.234914:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827107.234919:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827107.234933:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332860032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354302:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354311:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827112.354315:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354324:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354335:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354340:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827112.354376:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332860160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354410:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827112.354412:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354416:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354419:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354423:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354433:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827112.354453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332860288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827112.354466:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354469:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354475:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827112.354489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332860416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354497:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827112.354501:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354504:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354507:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354511:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354515:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827112.354524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332860544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827112.354538:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354542:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354545:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354548:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354552:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827112.354562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332860672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827112.354580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354584:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354587:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354590:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827112.354604:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332860800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827112.354617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354626:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827112.354640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332860928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827112.354649:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827112.354652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827112.354653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827112.354656:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8923 00000100:00080000:0.0:1787827112.354659:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827112.354662:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827112.354666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827112.354676:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332861056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827112.738444:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827115.426509:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827115.426551:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827115.426568:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426609:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426617:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426625:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426631:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426638:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426644:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827115.426650:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787827116.834427:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827117.474402:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.474412:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827117.474416:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.474425:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.474434:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.474440:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827117.474472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332861184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.474504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.474507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827117.474509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.474513:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.474517:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.474521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.474529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827117.474541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332861312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.474550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.474553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827117.474555:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.474558:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.474561:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.474565:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.474569:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827117.474578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332861440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.474591:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.474594:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827117.474595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.474598:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.474601:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.474604:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.474609:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827117.474618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332861568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.474631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.474634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827117.474636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.474639:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.474642:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.474645:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.474649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827117.474658:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332861696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.475627:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.475630:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827117.475633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.475638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.475645:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.475650:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.475657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827117.475680:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332861824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.475690:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.475693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827117.475694:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.475697:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.475700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.475703:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.475707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827117.475718:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332861952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827117.475726:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827117.475728:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827117.475730:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827117.475733:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8928 00000100:00080000:0.0:1787827117.475736:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827117.475739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827117.475743:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827117.475753:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332862080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594357:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827122.594370:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594380:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594390:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594410:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827122.594448:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332862208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594475:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827122.594480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594484:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594488:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827122.594514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332862336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594524:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827122.594530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594535:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594541:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594546:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594552:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827122.594565:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332862464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827122.594579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594583:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594587:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594590:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827122.594610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332862592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827122.594634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594638:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594642:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594649:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827122.594674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332862720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594686:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594690:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827122.594694:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594704:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594710:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594720:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827122.594745:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332862848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827122.594766:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594771:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594775:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594779:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594784:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827122.594798:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332862976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827122.594808:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827122.594811:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827122.594813:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827122.594819:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8933 00000100:00080000:0.0:1787827122.594824:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827122.594832:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827122.594839:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827122.594868:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332863104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827122.978409:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827127.074465:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827127.714352:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714361:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827127.714365:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714373:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714390:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827127.714426:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332863232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714453:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827127.714455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714467:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714476:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827127.714489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332863360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827127.714510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714518:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714522:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827127.714542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332863488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827127.714556:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714565:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714570:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714575:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827127.714591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332863616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827127.714601:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714603:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714606:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714611:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827127.714628:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332863744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714642:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827127.714645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714654:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714657:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827127.714671:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332863872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714688:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827127.714692:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714697:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714704:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827127.714722:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332864000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827127.714732:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827127.714734:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827127.714735:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827127.714740:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8938 00000100:00080000:3.0:1787827127.714743:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827127.714747:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827127.714751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827127.714761:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332864128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834341:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834352:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827132.834357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834371:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834404:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827132.834459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332864256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834496:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827132.834498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834503:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834507:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834511:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827132.834538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332864384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834558:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827132.834566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834576:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834593:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827132.834617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332864512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834635:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827132.834641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834651:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834659:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834667:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834674:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827132.834690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332864640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834700:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834704:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827132.834706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834714:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834723:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834733:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834743:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827132.834759:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332864768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834797:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834809:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827132.834815:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834829:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834834:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834839:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834843:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827132.834859:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332864896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834884:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.834893:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827132.834898:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.834917:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.834934:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.834944:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.834954:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827132.834975:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332865024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827132.834994:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827132.835003:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827132.835009:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827132.835018:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8943 00000100:00080000:3.0:1787827132.835025:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827132.835036:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827132.835051:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827132.835080:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332865152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827133.218532:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827137.314429:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827143.458386:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827147.554487:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827148.194326:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194336:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827148.194339:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194349:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194359:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194366:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827148.194402:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332867328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194430:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194433:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827148.194435:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194438:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194441:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194444:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194450:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827148.194459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332867456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194469:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194471:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827148.194473:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194474:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194476:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194478:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194480:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827148.194484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332867584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194490:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827148.194491:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194493:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194495:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194497:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194500:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827148.194507:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332867712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827148.194519:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194522:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194525:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194529:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194533:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827148.194542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332867840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194551:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827148.194555:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194557:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194560:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194562:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194565:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827148.194571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332867968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194579:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827148.194581:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194583:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194585:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194586:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827148.194596:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332868096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827148.194618:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827148.194621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827148.194623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827148.194627:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8958 00000100:00080000:0.0:1787827148.194631:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827148.194635:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827148.194639:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827148.194649:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332868224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314351:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314362:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827153.314365:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314374:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827153.314427:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332868352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314453:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314456:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827153.314458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314462:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827153.314494:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332868480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827153.314507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314511:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314515:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314518:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827153.314534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332868608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314544:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827153.314549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314556:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314573:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827153.314594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332868736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827153.314623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314626:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314662:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314667:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827153.314690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332868864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314700:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314705:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827153.314706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314711:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314717:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314721:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827153.314741:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332868992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314751:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827153.314764:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314776:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314784:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314789:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827153.314823:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332869120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827153.314840:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827153.314847:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827153.314851:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827153.314861:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 8964 00000100:00080000:0.0:1787827153.314872:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827153.314881:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827153.314888:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827153.314907:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332869248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827153.698410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827157.795273:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827163.938438:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827168.034418:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827174.178430:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827178.274414:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827181.986414:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827181.986463:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827181.986476:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986502:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986508:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986513:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986518:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986523:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986530:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827181.986540:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787827184.418382:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827188.514370:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827194.658389:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827198.754398:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827204.898529:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827208.994412:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827211.042516:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827215.138541:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827219.234510:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827219.874346:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874355:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827219.874359:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874370:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874380:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874388:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827219.874433:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332881664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827219.874474:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874478:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874482:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874496:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827219.874511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332881792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827219.874538:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874547:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874552:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874564:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827219.874596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332881920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874608:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827219.874614:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874623:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874642:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827219.874659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332882048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874679:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827219.874681:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874693:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874699:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874709:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827219.874746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332882176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874780:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874787:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827219.874790:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874799:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874804:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874825:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874832:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827219.874862:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332882304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.874923:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.874939:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827219.874945:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.874955:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.874962:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.874975:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.874991:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827219.875019:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332882432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827219.875069:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.875081:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827219.875090:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827219.875108:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9030 00000100:00080000:0.0:1787827219.875122:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827219.875135:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827219.875143:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827219.875165:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332882560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827221.282494:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827224.994338:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827224.994349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994357:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994366:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994373:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827224.994415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332882688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994439:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827224.994443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994446:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994448:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994451:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994458:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827224.994467:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332882816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994474:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994476:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827224.994477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994480:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994482:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994484:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994487:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827224.994496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332882944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827224.994510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994513:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994515:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994517:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994520:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827224.994530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332883072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994539:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827224.994540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994543:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994546:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994552:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827224.994564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332883200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827224.994577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994580:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994583:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994585:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994588:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827224.994596:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332883328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827224.994605:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994609:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994612:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994620:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994628:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827224.994638:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332883456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827224.994658:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827224.994664:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827224.994665:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827224.994672:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9035 00000100:00080000:0.0:1787827224.994680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827224.994689:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827224.994698:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827224.994715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332883584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827225.378389:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827229.474380:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827230.114302:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114309:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827230.114313:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114320:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114330:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114337:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827230.114375:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332883712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114408:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114411:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827230.114413:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114416:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114419:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114422:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114430:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827230.114442:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332883840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114453:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827230.114458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114465:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827230.114476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332883968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827230.114487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114489:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114491:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114494:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827230.114507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332884096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827230.114519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114525:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114528:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827230.114538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332884224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114544:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827230.114547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114552:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114555:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114561:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827230.114586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332884352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827230.114617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114637:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827230.114666:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332884480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827230.114705:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827230.114711:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827230.114716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827230.114726:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9040 00000100:00080000:0.0:1787827230.114733:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827230.114739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827230.114749:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827230.114763:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332884608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827231.522397:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827235.618454:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827239.714413:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827241.762428:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827245.858360:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827248.546641:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827248.546697:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827248.546721:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546771:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546783:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546798:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546809:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546818:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546829:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827248.546839:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787827249.954458:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827252.002427:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827255.714380:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714393:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827255.714399:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714413:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714427:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714437:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827255.714494:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332888832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714533:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827255.714535:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714540:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714545:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714550:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714563:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827255.714589:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332888960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714609:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827255.714611:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714615:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714619:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714625:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714631:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827255.714645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332889088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714660:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827255.714664:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714673:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714677:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714682:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714688:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827255.714701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332889216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714712:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714716:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827255.714718:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714722:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714729:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714738:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714748:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827255.714765:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332889344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714783:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714790:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827255.714795:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714801:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714806:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714811:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714815:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827255.714831:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332889472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714843:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714847:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827255.714849:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714853:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714858:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714866:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714874:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827255.714892:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332889600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827255.714940:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827255.714950:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827255.714956:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827255.714965:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9066 00000100:00080000:0.0:1787827255.714970:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827255.714975:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827255.714984:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827255.715002:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332889728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827256.098451:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827258.146466:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827260.194386:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827262.242474:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827265.954417:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827265.954430:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954456:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954467:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954474:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827265.954528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332890880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827265.954554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954557:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954560:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954570:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827265.954579:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332891008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827265.954597:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954601:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954606:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954611:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954616:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827265.954630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332891136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954648:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827265.954651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954657:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954662:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954667:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954671:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827265.954683:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332891264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954692:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954698:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827265.954701:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954707:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954712:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954717:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954724:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827265.954733:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332891392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954773:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827265.954782:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954789:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954794:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954799:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954801:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827265.954808:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332891520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954814:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954816:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827265.954818:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954820:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954823:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954827:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954833:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827265.954842:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332891648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827265.954875:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827265.954879:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827265.954880:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827265.954886:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9076 00000100:00080000:3.0:1787827265.954892:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827265.954899:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827265.954905:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827265.954916:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332891776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827266.338407:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827268.386373:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827270.434417:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827272.482465:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827276.578441:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827278.626378:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827280.674415:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827282.722401:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827286.818410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827288.866501:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827290.914440:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827292.962472:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827296.674368:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827296.674382:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674392:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674403:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674410:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827296.674449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332897024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674476:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674480:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827296.674481:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674485:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674488:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674491:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674499:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827296.674510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332897152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827296.674526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674530:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674534:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674538:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827296.674551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332897280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674566:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827296.674569:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674579:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674588:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674593:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827296.674610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332897408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674618:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827296.674623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674626:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674660:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674664:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674669:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827296.674682:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332897536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827296.674703:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674707:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674711:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674714:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827296.674728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332897664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674737:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674740:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827296.674743:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674753:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674762:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674782:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827296.674803:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332897792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827296.674829:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827296.674835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827296.674845:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827296.674855:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9107 00000100:00080000:0.0:1787827296.674866:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827296.674872:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827296.674877:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827296.674896:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332897920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827297.058518:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827299.106419:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827301.154441:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827301.794357:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794372:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827301.794389:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794402:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794416:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794425:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827301.794476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332898048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794508:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827301.794513:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794520:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794524:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827301.794548:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332898176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827301.794564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794571:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794578:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794582:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827301.794597:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332898304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794621:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794625:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827301.794627:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794631:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794644:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794648:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794654:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827301.794667:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332898432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794677:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794681:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827301.794683:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794691:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794695:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827301.794717:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332898560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794731:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794737:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827301.794742:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794751:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794759:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794767:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794774:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827301.794785:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332898688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794803:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827301.794810:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794815:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794819:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794827:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794834:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827301.794848:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332898816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827301.794886:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827301.794894:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827301.794897:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827301.794903:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9112 00000100:00080000:0.0:1787827301.794908:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827301.794912:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827301.794917:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827301.794934:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332898944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827303.202448:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827307.298438:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827309.346386:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827311.394371:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827313.442373:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827315.106474:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827315.106532:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827315.106549:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106584:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106594:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106602:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106611:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106620:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106630:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827315.106645:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00080000:0.0:1787827317.154323:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827317.154340:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154349:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154366:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827317.154397:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332901120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154419:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154421:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827317.154422:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154426:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154428:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154430:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154437:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827317.154445:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332901248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154452:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827317.154457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154467:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154470:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154472:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154476:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827317.154484:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332901376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154493:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154497:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827317.154500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154532:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154537:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827317.154563:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332901504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154577:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827317.154578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154583:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154607:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154614:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827317.154638:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332901632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154661:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827317.154669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154675:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154679:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154717:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827317.154753:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332901760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154804:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154816:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827317.154819:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154837:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154852:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154868:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154883:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827317.154910:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332901888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827317.154956:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827317.154961:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827317.154964:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827317.154971:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9127 00000100:00080000:0.0:1787827317.154975:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827317.154983:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827317.154994:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827317.155028:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332902016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827317.538385:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827319.586633:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827321.634465:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827322.274339:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274349:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827322.274354:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274376:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274384:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827322.274426:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332902144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274452:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827322.274456:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274464:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274482:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827322.274506:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332902272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827322.274519:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274522:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274525:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274527:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827322.274538:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332902400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274545:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827322.274548:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274553:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274557:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274562:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827322.274574:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332902528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274581:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827322.274584:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274587:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274589:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274592:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827322.274609:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332902656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827322.274640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274650:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274655:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274662:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274667:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827322.274678:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332902784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274688:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274691:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827322.274693:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274698:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274705:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274714:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827322.274738:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332902912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827322.274748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827322.274752:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827322.274756:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827322.274760:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9133 00000100:00080000:3.0:1787827322.274763:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827322.274769:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827322.274776:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827322.274787:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332903040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827323.682423:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827327.778504:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827329.826438:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827331.874456:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827332.514430:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514441:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827332.514446:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514456:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514468:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514476:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827332.514524:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332904192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514554:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514558:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827332.514561:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514565:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514569:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514574:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514585:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827332.514603:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332904320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514618:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514622:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827332.514624:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514629:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514633:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514637:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514643:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827332.514661:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332904448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514675:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514679:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827332.514681:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514686:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514690:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514694:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514702:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827332.514722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332904576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514756:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514765:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827332.514770:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514777:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514782:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514786:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514794:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827332.514812:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332904704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514832:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514838:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827332.514842:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514851:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514855:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514863:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514874:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827332.514893:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332904832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514917:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.514921:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827332.514924:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.514931:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.514939:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.514946:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.514958:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827332.514978:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332904960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827332.514998:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827332.515005:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827332.515008:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827332.515017:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9143 00000100:00080000:3.0:1787827332.515026:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827332.515036:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827332.515048:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827332.515069:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332905088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827333.922528:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827337.634387:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634394:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827337.634397:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634406:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634415:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634421:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827337.634455:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332905216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634485:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827337.634489:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634491:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634494:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634496:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.634512:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332905344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634519:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634521:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.634522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634527:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634529:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.634541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332905472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634548:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634550:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827337.634552:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634554:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634557:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634559:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634563:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827337.634570:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332905600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634576:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634578:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.634580:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634582:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634587:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634589:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634592:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827337.634601:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332905728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634610:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634613:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827337.634615:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634619:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634627:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634630:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634633:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.634642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332905856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634670:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.634673:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634677:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634680:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634682:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634685:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.634695:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332905984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.634712:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.634716:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.634719:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.634726:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9148 00000100:00080000:3.0:1787827337.634732:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.634736:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827337.634740:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827337.634752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332906112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827338.018492:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827340.066412:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827342.114439:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827342.754417:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754430:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827342.754436:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754449:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754463:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754472:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827342.754523:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332906240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754565:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827342.754576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754581:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754585:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754590:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754601:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827342.754617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332906368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827342.754639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754644:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754652:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827342.754678:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332906496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754696:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754700:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827342.754702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754707:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754712:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754716:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754723:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827342.754740:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332906624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754752:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754756:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827342.754759:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754764:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754768:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754772:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754778:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827342.754796:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332906752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754811:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754815:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827342.754818:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754823:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754828:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754832:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754839:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827342.754851:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332906880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.754864:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.754869:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827342.754872:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.754884:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.754893:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.754904:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.754916:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827342.754938:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332907008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827342.755002:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827342.755008:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827342.755012:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827342.755027:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9153 00000100:00080000:0.0:1787827342.755032:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827342.755043:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827342.755057:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827342.755095:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332907136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827344.162382:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827347.874330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874338:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827347.874342:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874359:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874365:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827347.874405:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332907264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874429:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827347.874430:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874435:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874438:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874445:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827347.874454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332907392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874462:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874466:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827347.874468:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874476:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874479:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874482:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827347.874495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332907520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874501:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874505:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827347.874506:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874508:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874511:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874513:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874516:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827347.874522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332907648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827347.874531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874533:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874536:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874542:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827347.874556:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332907776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874565:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827347.874567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874570:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874572:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874574:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874578:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827347.874584:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332907904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874590:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874593:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827347.874595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874600:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874603:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874611:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874620:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827347.874637:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332908032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827347.874656:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827347.874662:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827347.874666:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827347.874675:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9158 00000100:00080000:3.0:1787827347.874679:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827347.874683:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827347.874688:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827347.874713:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332908160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827348.258487:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827350.306443:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827352.354396:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827352.994361:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994369:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827352.994373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994381:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994390:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994397:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827352.994431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332908288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827352.994458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994465:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.994486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332908416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.994502:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994506:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994510:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994517:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.994541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332908544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827352.994562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994565:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994573:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827352.994588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332908672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.994600:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994602:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994605:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994607:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827352.994618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332908800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994625:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994627:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827352.994629:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994634:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994639:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.994659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332908928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994670:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.994673:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994682:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994687:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994692:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994695:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.994708:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332909056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.994723:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.994727:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.994729:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.994732:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9163 00000100:00080000:0.0:1787827352.994734:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.994737:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827352.994740:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827352.994752:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332909184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827354.402497:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827358.498443:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827360.546403:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827362.594475:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827364.642444:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827368.354325:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354334:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827368.354338:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354346:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354354:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354359:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827368.354392:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332911360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354413:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354415:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827368.354417:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354421:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354423:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354426:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354435:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827368.354444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332911488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354455:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354458:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827368.354459:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354467:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354470:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827368.354479:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332911616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354490:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354492:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827368.354494:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354497:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354499:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354502:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827368.354513:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332911744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354526:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354529:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827368.354530:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354535:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354538:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354541:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827368.354551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332911872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354562:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827368.354567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354578:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354585:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354593:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827368.354617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332912000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354629:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827368.354630:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354633:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354638:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354647:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354653:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827368.354681:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332912128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827368.354717:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827368.354725:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827368.354728:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827368.354734:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9179 00000100:00080000:3.0:1787827368.354739:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827368.354747:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827368.354753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827368.354771:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332912256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827368.738531:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827370.786506:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827372.834432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827373.474311:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474320:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827373.474323:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474332:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474342:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474348:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827373.474384:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332912384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474410:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474413:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827373.474414:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474418:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474420:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474423:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474431:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827373.474440:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332912512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474449:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827373.474456:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474477:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827373.474497:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332912640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474520:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827373.474522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474526:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474531:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474535:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474540:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827373.474551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332912768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827373.474562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474567:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474569:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474573:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827373.474580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332912896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474589:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827373.474591:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474594:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474597:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474599:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827373.474607:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332913024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474613:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474615:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827373.474616:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474618:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474620:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474623:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827373.474631:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332913152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827373.474637:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827373.474639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827373.474640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827373.474642:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9184 00000100:00080000:3.0:1787827373.474644:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827373.474646:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827373.474648:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827373.474655:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332913280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827374.882428:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827378.594323:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594331:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827378.594334:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594342:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594357:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827378.594389:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332913408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594414:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594417:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827378.594418:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594421:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594430:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594440:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827378.594449:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332913536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594460:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827378.594461:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594464:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594468:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594473:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827378.594495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332913664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594510:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827378.594512:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594517:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594529:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594534:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594539:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827378.594549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332913792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594558:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827378.594560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594563:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594566:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594568:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827378.594580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332913920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827378.594608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594612:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594615:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594618:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594622:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827378.594635:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332914048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594645:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594648:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827378.594650:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594653:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594658:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594661:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594664:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827378.594676:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332914176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827378.594685:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827378.594687:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827378.594689:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827378.594691:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9189 00000100:00080000:0.0:1787827378.594695:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827378.594701:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827378.594705:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827378.594718:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332914304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827378.978556:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827381.026453:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827383.074400:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827383.714422:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714432:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827383.714435:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714445:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714455:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714461:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827383.714496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332914432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714533:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714536:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827383.714538:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714541:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714544:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714546:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714554:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.714563:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332914560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714577:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.714578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714581:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714583:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714588:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.714613:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332914688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714628:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827383.714636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714642:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714655:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714661:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714668:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827383.714687:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332914816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714701:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714706:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.714708:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714714:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714718:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714724:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714731:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827383.714749:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332914944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714764:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714772:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827383.714777:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714789:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714795:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714807:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714817:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.714841:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332915072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714856:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714860:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.714863:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714868:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714875:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714885:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714893:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.714911:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332915200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.714932:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.714944:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.714950:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.714963:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9194 00000100:00080000:3.0:1787827383.714974:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.714985:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827383.714996:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827383.715012:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332915328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827385.122414:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827388.834370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827388.834381:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834389:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834398:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834404:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827388.834474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332915456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.834495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834497:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827388.834498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834501:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834506:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.834514:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827388.834525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332915584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.834569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827388.834579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834588:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834611:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834635:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.834647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827388.834674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332915712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.834724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834731:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827388.834735:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834743:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834752:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834759:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.834782:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827388.834804:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332915840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.834821:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834826:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827388.834828:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834837:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834842:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834868:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.834879:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827388.834902:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332915968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.834929:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.834938:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827388.834959:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.834969:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.834976:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.834989:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.835001:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827388.835071:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332916096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.835101:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.835108:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827388.835116:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.835133:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.835141:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.835154:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.835167:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827388.835204:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332916224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827388.835227:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827388.835233:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827388.835238:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827388.835245:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9199 00000100:00080000:0.0:1787827388.835266:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827388.835301:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827388.835305:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827388.835316:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332916352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827389.218418:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827391.266588:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827393.314466:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827395.362364:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827399.458483:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827401.506454:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827403.554428:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827404.194364:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194372:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827404.194375:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194383:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194392:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194399:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827404.194431:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332918528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194453:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194456:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827404.194457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194463:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194466:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827404.194481:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332918656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194490:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827404.194499:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194506:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194512:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827404.194524:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332918784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827404.194534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194537:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194547:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827404.194565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332918912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827404.194577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194579:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194581:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194584:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827404.194596:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332919040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194628:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827404.194637:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194645:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194648:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194653:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194657:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827404.194667:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332919168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194674:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194676:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827404.194678:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194683:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194685:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194690:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194695:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827404.194701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332919296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827404.194708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827404.194710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827404.194711:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827404.194713:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9214 00000100:00080000:3.0:1787827404.194723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827404.194728:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827404.194733:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827404.194748:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332919424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827405.602505:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827409.314367:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827409.314412:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314423:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314436:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314444:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827409.314506:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332919552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.314554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314558:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827409.314561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314582:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314585:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.314597:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827409.314617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332919680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.314629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827409.314655:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314664:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314677:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.314682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827409.314698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332919808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.314709:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314714:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827409.314716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314719:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314724:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314729:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.314735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827409.314750:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332919936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.314763:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827409.314769:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314773:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314776:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314781:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.314787:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827409.314802:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332920064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.314844:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.314892:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827409.314899:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.314934:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.314944:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.314956:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.314968:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827409.315010:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332920192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.315046:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.315051:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827409.315054:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.315062:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.315073:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.315079:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.315087:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827409.315107:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332920320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827409.315127:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827409.315139:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827409.315144:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827409.315152:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9220 00000100:00080000:0.0:1787827409.315159:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827409.315173:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827409.315179:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827409.315195:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332920448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827409.698549:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827411.746534:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827413.794507:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827414.434406:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434418:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827414.434424:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434436:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434450:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434459:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827414.434511:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663332920576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.434549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434554:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827414.434557:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434562:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434577:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434583:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.434595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827414.434611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663332920704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.434627:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827414.434654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434669:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434676:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434680:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.434685:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827414.434702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663332920832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.434715:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434720:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827414.434723:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434727:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434751:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434756:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.434762:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827414.434780:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663332920960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.434795:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434800:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827414.434803:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434809:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434812:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434816:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.434847:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827414.434880:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663332921088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.434935:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.434941:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827414.434945:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.434955:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.434963:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.434977:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.434993:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827414.435020:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663332921216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.435048:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.435053:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827414.435056:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.435062:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.435066:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.435075:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.435091:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827414.435119:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663332921344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827414.435153:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827414.435166:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827414.435173:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827414.435186:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9225 00000100:00080000:3.0:1787827414.435201:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827414.435294:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827414.435314:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827414.435344:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663332921472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827415.842510:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827419.554341:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554352:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827419.554356:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554382:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554394:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554403:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827419.554446:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663332921600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554482:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827419.554490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554497:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554503:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554510:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.554523:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827419.554561:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663332921728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827419.554599:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554604:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554608:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554612:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.554635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827419.554650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663332921856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554660:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554664:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827419.554669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554675:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554682:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554686:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.554692:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827419.554704:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332921984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554734:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827419.554742:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554754:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554765:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554776:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.554813:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827419.554854:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332922112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554880:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554890:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827419.554895:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554909:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554919:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.554926:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.554932:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827419.554944:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332922240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.554954:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.554959:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827419.554964:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.554971:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.554992:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.555002:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.555010:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827419.555044:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332922368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827419.555105:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827419.555120:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827419.555126:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827419.555137:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9230 00000100:00080000:3.0:1787827419.555147:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827419.555160:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827419.555185:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827419.555199:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332922496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827419.938592:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827421.986561:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827424.034600:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827424.674395:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674416:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827424.674423:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674440:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674457:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674468:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827424.674549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332922624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.674628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674636:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827424.674641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674650:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674656:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674662:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.674677:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827424.674700:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332922752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.674718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827424.674727:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674737:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674781:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674788:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.674797:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827424.674819:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332922880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.674834:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674839:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827424.674841:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674847:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674853:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674857:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.674864:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827424.674877:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332923008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.674890:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674899:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827424.674903:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674917:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674921:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674926:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.674933:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827424.674948:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332923136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.674960:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.674965:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827424.674968:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.674974:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.674979:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.674984:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.674990:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827424.675009:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332923264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.675077:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.675086:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827424.675092:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.675129:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.675140:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.675162:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.675187:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827424.675222:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332923392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827424.675293:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827424.675299:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827424.675319:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827424.675325:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9235 00000100:00080000:0.0:1787827424.675333:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827424.675340:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827424.675351:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827424.675403:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332923520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827426.082451:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827429.794294:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794307:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827429.794312:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794325:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794339:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794348:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827429.794395:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332923648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794420:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794423:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827429.794424:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794428:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794431:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794435:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827429.794454:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332923776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827429.794470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794480:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794482:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827429.794494:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332923904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827429.794504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794507:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794515:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827429.794522:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332924032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827429.794531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794538:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827429.794551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332924160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794566:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827429.794567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794570:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794574:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827429.794584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332924288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827429.794605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794612:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794615:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794618:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827429.794631:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332924416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827429.794642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827429.794647:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827429.794650:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827429.794660:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9240 00000100:00080000:0.0:1787827429.794665:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827429.794669:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827429.794671:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827429.794678:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332924544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827430.178437:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827432.226400:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827434.274487:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827434.914356:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827434.914371:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914383:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914396:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914404:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827434.914451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332924672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914477:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914481:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827434.914483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914489:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914494:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914499:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914513:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827434.914535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332924800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827434.914551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914554:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914557:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914560:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914565:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827434.914574:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332924928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827434.914585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914591:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914600:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914604:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827434.914612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332925056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827434.914623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914626:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914630:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914633:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827434.914647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332925184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914676:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914679:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827434.914680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914702:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914709:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914714:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914737:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827434.914757:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332925312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914768:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827434.914773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914777:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914781:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914788:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914794:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827434.914828:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332925440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827434.914881:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827434.914895:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827434.914901:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827434.914913:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9245 00000100:00080000:0.0:1787827434.914924:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827434.914956:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827434.914977:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827434.915022:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332925568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827436.322456:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827440.034324:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034332:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827440.034336:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034346:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034355:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034363:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827440.034400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332925696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034430:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.034432:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034458:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.034483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332925824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.034496:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034499:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034502:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034504:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.034517:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332925952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.034530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034532:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034534:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034536:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034540:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.034551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332926080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.034588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034594:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034599:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034605:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034609:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.034623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332926208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034645:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.034661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034674:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034677:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034681:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.034695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332926336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034714:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.034718:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034723:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034728:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034736:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.034759:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332926464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.034772:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.034776:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.034777:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.034782:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9250 00000100:00080000:0.0:1787827440.034786:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.034793:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827440.034798:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.034813:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332926592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827440.418464:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827442.466487:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827444.514457:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827446.562474:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827448.226567:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827448.226627:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827448.226649:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226696:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226706:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226716:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226727:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226743:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226771:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827448.226787:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00080000:0.0:1787827450.274381:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827450.274391:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274399:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274408:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274425:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827450.274453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332927744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274488:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827450.274492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274495:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274498:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274500:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827450.274514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332927872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827450.274526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274529:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274531:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274533:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274544:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827450.274552:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332928000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274564:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274566:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827450.274567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274572:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274574:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274644:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827450.274656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332928128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827450.274667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274670:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274673:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274678:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827450.274702:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332928256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274714:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827450.274727:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274735:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274741:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274751:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274761:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827450.274779:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332928384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274790:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274794:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827450.274796:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274804:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274809:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274813:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274818:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827450.274832:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332928512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827450.274851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827450.274857:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827450.274861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827450.274871:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9261 00000100:00080000:0.0:1787827450.274877:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827450.274882:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827450.274887:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827450.274908:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332928640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827450.658420:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827452.706406:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827454.754379:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827456.802361:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827460.514342:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827460.514357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514378:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514385:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827460.514418:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332929792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514440:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827460.514444:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514459:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827460.514467:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332929920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514475:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827460.514479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514482:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514484:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514489:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827460.514503:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332930048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827460.514530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514533:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514540:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514544:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827460.514554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332930176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827460.514564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514568:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827460.514579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332930304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514587:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827460.514588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514590:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514592:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514594:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514597:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827460.514607:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332930432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827460.514618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514625:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827460.514637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332930560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827460.514644:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827460.514647:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827460.514649:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827460.514653:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9271 00000100:00080000:0.0:1787827460.514658:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827460.514664:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827460.514673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827460.514689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332930688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827460.898367:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827462.946519:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827464.994461:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827465.634323:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634332:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827465.634335:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634344:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634363:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634375:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827465.634408:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332930816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827465.634436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634439:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634453:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827465.634464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332930944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634499:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827465.634503:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634507:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634510:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634521:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827465.634540:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332931072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827465.634565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634578:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634582:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634588:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827465.634603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332931200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634624:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827465.634628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634637:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634643:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634648:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634656:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827465.634668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332931328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634676:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634680:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827465.634682:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634698:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634703:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634709:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827465.634732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332931456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634750:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634759:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827465.634763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634770:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634776:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634784:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634790:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827465.634804:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332931584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827465.634815:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827465.634819:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827465.634822:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827465.634825:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9276 00000100:00080000:0.0:1787827465.634828:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827465.634830:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827465.634833:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827465.634840:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332931712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827467.042384:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827471.138417:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827473.186385:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827475.234399:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827477.282627:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827481.378445:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827483.426437:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827485.474395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827487.522455:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827491.234413:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234428:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827491.234435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234451:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234468:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234481:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827491.234557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332935936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234616:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234620:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827491.234622:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234627:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234631:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234636:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827491.234662:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332936064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234673:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827491.234679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234683:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234686:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234693:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827491.234737:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332936192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234750:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234755:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827491.234758:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234763:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234781:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234785:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234790:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827491.234812:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332936320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234822:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234825:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827491.234827:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234831:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234833:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234836:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234839:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827491.234849:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332936448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234856:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234859:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827491.234861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234864:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234866:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234873:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234879:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827491.234897:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332936576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.234931:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.234939:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827491.234942:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.234950:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.234957:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.234965:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.234977:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827491.234994:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332936704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827491.235014:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827491.235026:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827491.235030:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827491.235038:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9301 00000100:00080000:0.0:1787827491.235044:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827491.235053:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827491.235059:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827491.235078:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332936832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827491.618596:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827493.666408:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827495.714369:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827497.762469:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827501.858387:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827503.906473:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827505.954585:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827506.594328:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827506.594340:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594357:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594362:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827506.594394:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332939008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594415:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594417:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827506.594419:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594422:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594424:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594426:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594444:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827506.594452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332939136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594458:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594460:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.594462:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594464:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594472:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827506.594480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332939264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594486:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594489:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827506.594490:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594505:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594507:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594519:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827506.594526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332939392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.594536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594542:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594544:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594547:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827506.594554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332939520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594570:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827506.594574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594579:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594585:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594591:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827506.594600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332939648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594606:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.594609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594611:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594614:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594616:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594619:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827506.594624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332939776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.594632:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.594639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.594642:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.594650:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9317 00000100:00080000:0.0:1787827506.594654:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.594657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827506.594662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827506.594674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332939904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827508.002362:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827511.714299:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714307:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827511.714311:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714325:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714334:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714340:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827511.714372:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332940032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714400:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714404:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827511.714406:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714411:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714415:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714419:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714429:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.714441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332940160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827511.714457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714465:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714470:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714474:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.714492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332940288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827511.714509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714513:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714518:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714521:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827511.714536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332940416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714542:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714545:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827511.714546:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714548:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714553:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714558:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827511.714578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332940544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827511.714612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714619:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714627:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.714647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332940672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827511.714664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714667:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714669:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.714685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332940800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.714693:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.714699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827511.714703:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.714706:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9322 00000100:00080000:0.0:1787827511.714711:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.714714:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827511.714717:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827511.714729:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332940928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827512.098363:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827514.146430:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827516.194409:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827518.242456:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827521.954344:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954352:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827521.954356:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827521.954410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332942080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827521.954436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954439:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954441:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954444:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827521.954459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332942208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954469:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827521.954470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827521.954484:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332942336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954494:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827521.954495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954500:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954503:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954505:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954509:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827521.954516:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332942464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827521.954549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954553:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954556:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954557:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827521.954576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332942592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827521.954591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954596:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954612:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954617:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827521.954648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332942720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827521.954661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954664:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954669:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954676:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827521.954686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332942848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827521.954761:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827521.954770:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827521.954776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827521.954793:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9332 00000100:00080000:0.0:1787827521.954808:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827521.954824:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827521.954837:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827521.954860:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332942976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827522.338369:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827524.386486:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827526.434443:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827527.074300:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074308:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827527.074311:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074320:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074328:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827527.074377:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332943104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074403:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074406:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.074407:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074411:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074413:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074415:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074422:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.074430:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332943232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.074443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074459:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.074472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332943360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.074500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074504:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074508:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074511:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.074527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332943488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.074542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074548:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074552:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074555:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074559:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.074568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332943616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074577:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.074578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074580:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074582:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074591:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.074613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332943744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074632:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.074640:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074646:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074653:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074658:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.074677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332943872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.074688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.074691:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.074693:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.074698:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9337 00000100:00080000:0.0:1787827527.074703:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.074710:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827527.074718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.074730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332944000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827528.482406:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827532.194352:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194362:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827532.194367:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194379:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194391:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194399:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827532.194444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332944128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194476:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.194477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194481:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194483:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194486:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194495:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.194506:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332944256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.194522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194528:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194532:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194541:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.194579:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332944384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194600:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.194605:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194608:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194612:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194616:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194623:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.194638:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332944512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194653:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.194659:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194663:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194667:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194672:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194678:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.194690:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332944640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194701:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.194703:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194706:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194708:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194711:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.194739:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332944768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194755:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194780:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.194785:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194808:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194815:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194823:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194840:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.194857:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332944896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.194872:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.194881:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.194883:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.194887:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9342 00000100:00080000:0.0:1787827532.194890:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.194893:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827532.194897:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.194907:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332945024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827532.578377:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827534.626485:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827536.674482:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827537.314319:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314331:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827537.314336:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314347:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314360:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314370:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827537.314437:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332945152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314469:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314484:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827537.314486:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314492:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314496:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314500:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314512:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827537.314532:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332945280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314545:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827537.314546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314549:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314552:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314557:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827537.314572:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332945408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827537.314585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314588:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314590:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314593:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827537.314605:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332945536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314611:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314614:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827537.314615:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314618:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314621:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314623:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827537.314634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332945664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314656:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314659:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827537.314660:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314664:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314667:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314670:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314672:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827537.314679:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332945792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314686:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314688:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827537.314689:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314692:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314694:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314697:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314700:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827537.314718:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332945920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827537.314725:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827537.314728:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827537.314731:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827537.314736:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9348 00000100:00080000:0.0:1787827537.314740:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827537.314745:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827537.314749:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827537.314756:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332946048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827538.722410:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827542.435014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827542/real 1787827542] req@ffff9057d0de0e00 x1874663332946816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827542.435022:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332946816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827542.435028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827542.435029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827542.435031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332946816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827542.435043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332946944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827542.435053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827542.435059:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827542/real 1787827542] req@ffff9057d0de3800 x1874663332946944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827542.435066:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332946944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827542.435071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827542.435073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827542.435075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332946944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827542.435078:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827542.435088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827542.435093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827542/real 1787827542] req@ffff9057d0de0000 x1874663332947072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827542.435099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332947072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827542.435109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827542.435111:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827542.435114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947072:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827542.818406:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827543.842415:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827544.866367:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827545.890463:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827546.914466:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827547.554405:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554416:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827547.554420:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554432:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554444:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554452:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827547.554498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332947200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827547.554526:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554529:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827547.554531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554534:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827547.554560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332947328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827547.554569:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827547.554574:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827547.554577:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332947200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827547.554579:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554585:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554588:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554591:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827547.554599:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332947456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827547.554606:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554609:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827547.554611:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554614:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554616:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554618:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827547.554628:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332947584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827547.554634:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827547.554638:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554641:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554643:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554646:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827547.554656:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332947712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827547.554662:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827547.554669:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554672:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827547.554674:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554678:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554681:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554684:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554687:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827547.554693:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332947840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827547.554698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554700:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827547.554702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827547.554706:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:0.0:1787827547.554711:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827547.554715:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554721:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827547.554722:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158f100 x1874663332947200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827547.554738:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332947968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827547.554748:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663332947200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827547.554753:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827547.554761:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827547.554763:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827547.554766:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827547.554770:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9358 00000100:00080000:1.0:1787827547.554776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827547.554778:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827547.554784:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332947200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827547.554786:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827547.554795:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827547.554810:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332948096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827547.554820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827547.554849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.554861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158e300 x1874663332947328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.554871:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663332947328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.554877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.554880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.554882:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827547.554887:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827547.554899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.554921:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158dc00 x1874663332947456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.554929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663332947456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.554935:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.554938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.554940:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827547.554944:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827547.554959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.554965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158c000 x1874663332947584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.554972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663332947584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.554978:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.554979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.554981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827547.554985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827547.554998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.555003:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158df80 x1874663332947712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.555011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663332947712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.555016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.555018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.555020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332947712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827547.555023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827547.555034:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.555040:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158e680 x1874663332947840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.555047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663332947840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.555052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.555054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.555056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827547.555059:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827547.555069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.555075:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158f800 x1874663332947968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.555094:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663332947968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.555100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.555102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.555104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332947968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827547.555123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827547.555135:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827547.555141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827547/real 1787827547] req@ffff9057f158ca80 x1874663332948096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827547.555149:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663332948096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827547.555154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827547.555156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827547.555158:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948096:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827547.938530:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827548.962523:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827549.986542:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827551.010457:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827552.674358:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674369:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827552.674374:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.674385:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.674396:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.674404:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827552.674489:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332948224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.674518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674523:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827552.674525:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.674530:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.674534:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.674540:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827552.674551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827552.674567:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332948352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827552.674573:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332948224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827552.674588:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674592:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.674594:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.674599:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.674603:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.674624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827552.674630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827552.674644:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332948480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.674655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674659:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827552.674664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827552.674665:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.674673:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.674678:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.674683:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827552.674689:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827552.674703:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332948608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827552.674730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158ca80 x1874663332948224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827552.674731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674741:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.674755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827552.674760:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663332948224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827552.674764:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.674772:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827552.674793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827552.674796:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787827552.674800:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.674805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332948224:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787827552.674806:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827552.674831:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827552.674834:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332948736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.674850:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674856:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787827552.674856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827552.674862:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.674884:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00100000:1.0:1787827552.674889:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158f800 x1874663332948352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827552.674892:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.674911:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827552.674919:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827552.674920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663332948352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.674931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.674934:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787827552.674940:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332948864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827552.674940:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827552.674952:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948480:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787827552.674990:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.674995:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.674998:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.675005:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:0.0:1787827552.675011:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827552.675016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827552.675021:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827552.675029:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827552.675064:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158e680 x1874663332948480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827552.675072:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332948992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827552.675081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663332948480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827552.675091:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.675114:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.675118:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.675128:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9363 00000100:00080000:1.0:1787827552.675133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827552.675134:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.675144:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787827552.675145:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948480:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787827552.675156:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827552.675162:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1787827552.675193:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332949120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827552.675204:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827552.675218:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158df80 x1874663332948608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827552.675227:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663332948608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.675233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.675236:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675238:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827552.675251:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827552.675262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827552.675269:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158c000 x1874663332948736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827552.675276:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663332948736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.675306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.675308:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675328:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332948736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827552.675334:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827552.675356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827552.675368:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158dc00 x1874663332948864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827552.675382:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663332948864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.675398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.675400:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827552.675407:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827552.675422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827552.675465:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158e300 x1874663332948992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827552.675478:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663332948992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.675486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.675490:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675494:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332948992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827552.675500:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332949120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827552.675518:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827552.675530:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827552] req@ffff9057f158f100 x1874663332949120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827552.675542:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663332949120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827552.675551:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827552.675554:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827552.675558:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332949120:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827553.058389:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827554.082397:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827555.106429:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827556.130476:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827557.154396:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827557.794493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332949248:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827557.794553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.794621:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de0a80 x1874663332949248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.794642:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332949248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.794653:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.794681:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.794692:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332949248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827557.794730:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.794764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.794805:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de1500 x1874663332949376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.794862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332949376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.794875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.794880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.794885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.794894:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.794921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.794949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de1c00 x1874663332949504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.794959:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332949504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.794965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.794968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.794970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.794974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827557.794986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.795008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de1f80 x1874663332949632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.795025:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332949632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.795035:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.795037:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.795039:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827557.795042:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827557.795057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.795067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de0000 x1874663332949760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.795078:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332949760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.795097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.795100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.795104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332949760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827557.795119:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332949888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.795134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.795142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de3800 x1874663332949888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.795149:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332949888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.795154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.795156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.795159:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332949888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.795163:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.795173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.795181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de0e00 x1874663332950016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.795193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332950016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.795201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.795204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.795207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.795214:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827557.795294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827557.795304:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff9057d0de3100 x1874663332950144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.795313:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332950144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.795319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.795321:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827557.795323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950144:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827558.178386:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827559.202417:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827560.226371:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827561.250458:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827562.914494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332950272:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827562.914588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.914640:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de1180 x1874663332950272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.914680:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332950272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.914703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.914715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.914723:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332950272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827562.914754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.914808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.914831:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de2a00 x1874663332950400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.914865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332950400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.914874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.914879:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.914882:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.914888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.914922:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.914931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de0700 x1874663332950528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.914941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332950528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.914948:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.914950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.914952:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.914957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827562.914972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.914979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de1880 x1874663332950656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.914988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332950656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.914994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.915005:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.915008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827562.915012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827562.915026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.915033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de3100 x1874663332950784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.915043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332950784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.915055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.915058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.915061:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332950784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827562.915065:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.915096:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.915103:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de0e00 x1874663332950912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.915112:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332950912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.915118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.915120:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.915122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332950912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.915153:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.915168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.915176:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de3800 x1874663332951040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.915184:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332951040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.915190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.915192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.915195:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.915199:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827562.915211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827562.915218:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff9057d0de0000 x1874663332951168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.915244:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332951168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.915250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.915252:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827562.915254:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951168:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827563.298535:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827564.322481:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827565.346400:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827566.370353:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827567.394427:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827568.034334:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.034349:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827568.034366:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034378:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00080000:0.0:1787827568.034392:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034401:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827568.034454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332951296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.034488:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.034493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.034495:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034501:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00080000:0.0:1787827568.034505:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034510:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034533:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.034552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332951424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827568.034557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332951296:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827568.034601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.034605:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.034607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00080000:0.0:1787827568.034618:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034623:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.034645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332951552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.034657:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.034662:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.034664:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034672:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9378 00000800:00000100:1.0:1787827568.034673:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827568.034680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034687:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034695:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827568.034709:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332951680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.034721:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.034725:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.034728:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034734:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00100000:1.0:1787827568.034740:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158ce00 x1874663332951296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827568.034741:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034751:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034788:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827568.034804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663332951296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827568.034806:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332951808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827568.034848:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.034855:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827568.034858:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827568.034862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332951296:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827568.034865:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.034869:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034877:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00080000:0.0:1787827568.034882:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.034887:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034894:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827568.034916:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827568.034939:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332951936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.034955:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827568.034961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827568.034963:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.034967:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.034974:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00080000:0.0:1787827568.034980:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827568.034984:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158d880 x1874663332951424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827568.034989:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.034999:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827568.035009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663332951424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827568.035018:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332952064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827568.035023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951424:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787827568.035043:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.035052:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.035065:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.035086:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9378 00000100:00100000:1.0:1787827568.035091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787827568.035092:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.035101:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827568.035109:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827568.035124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787827568.035127:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332952192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827568.035141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158fb80 x1874663332951552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035157:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663332951552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035185:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827568.035191:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827568.035212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827568.035267:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158d180 x1874663332951680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035290:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663332951680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035318:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035322:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827568.035329:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827568.035352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827568.035364:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158c700 x1874663332951808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035388:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663332951808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035402:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035406:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332951808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827568.035426:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827568.035464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827568.035477:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158d500 x1874663332951936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035489:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663332951936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035520:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035524:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332951936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827568.035530:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332952064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827568.035566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827568.035587:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158ed80 x1874663332952064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035600:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663332952064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035609:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035613:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035617:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332952064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827568.035639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332952192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827568.035669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827568.035682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827568] req@ffff9057f158ea00 x1874663332952192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827568.035694:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663332952192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827568.035703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827568.035706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827568.035711:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332952192:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827568.418409:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827569.442422:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827570.466398:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827571.490470:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827573.154345:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154353:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827573.154355:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154364:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154373:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154379:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827573.154409:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663332952320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154436:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827573.154437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154440:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154442:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154444:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154451:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827573.154496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332952448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154511:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154515:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827573.154534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154541:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154543:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154546:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827573.154562:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332952576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154568:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154571:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827573.154572:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154575:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154577:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154579:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154601:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827573.154612:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332952704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154619:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154622:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827573.154623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154626:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154630:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154633:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154636:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827573.154644:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332952832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827573.154657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154660:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154662:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154667:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827573.154674:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332952960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154680:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154683:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827573.154684:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154686:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154689:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154691:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827573.154700:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332953088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827573.154708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827573.154711:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827573.154712:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827573.154715:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9383 00000100:00080000:1.0:1787827573.154717:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827573.154720:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827573.154753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827573.154761:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332953216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827573.538520:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827574.562409:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827575.586403:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827576.610414:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827577.634456:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827578.274314:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274324:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827578.274328:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274338:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274349:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274356:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827578.274395:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332953344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274442:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274446:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827578.274449:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274454:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274459:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274464:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827578.274486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332953472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274500:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274503:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827578.274505:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274508:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274511:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274514:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827578.274528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332953600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274537:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827578.274538:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274542:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274544:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274546:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827578.274558:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332953728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274572:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827578.274576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274580:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274582:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274585:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827578.274600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332953856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274614:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274617:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827578.274619:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274622:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274636:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274639:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274645:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827578.274658:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332953984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274666:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274673:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827578.274678:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274684:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274702:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274707:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274711:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827578.274725:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332954112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827578.274750:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827578.274753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827578.274754:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827578.274760:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9389 00000100:00080000:0.0:1787827578.274766:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827578.274770:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827578.274774:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827578.274785:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332954240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827578.658390:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827579.682455:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827580.706439:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827581.346470:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827581.346507:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827581.346521:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346555:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346562:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346567:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346572:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346578:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346586:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827581.346592:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:1.0:1787827581.730542:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827583.394384:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394398:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827583.394403:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394418:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394432:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394459:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827583.394502:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332954368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827583.394549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394555:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394560:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394565:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394575:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827583.394589:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332954496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394607:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827583.394609:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394614:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394619:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394631:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827583.394646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332954624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394674:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394679:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827583.394682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394687:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394692:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394700:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827583.394728:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332954752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394738:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394742:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827583.394744:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394769:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394776:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394787:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394795:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827583.394814:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332954880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394835:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394844:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827583.394849:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394855:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394860:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394866:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394884:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827583.394906:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332955008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.394933:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.394938:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827583.394943:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.394954:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.394962:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.394971:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.394983:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827583.395002:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332955136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827583.395026:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827583.395030:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827583.395033:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827583.395039:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9394 00000100:00080000:0.0:1787827583.395053:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827583.395064:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827583.395073:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827583.395093:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332955264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827583.778502:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827584.802584:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827585.826348:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827586.850515:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827587.874371:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827588.514361:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514372:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827588.514377:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514389:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514401:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514409:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827588.514480:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332955392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514509:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514512:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827588.514513:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514517:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514520:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514523:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827588.514544:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332955520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514552:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514555:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827588.514556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514559:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514560:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514567:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827588.514575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332955648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514599:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827588.514604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514607:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514610:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514614:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514634:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827588.514646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332955776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514653:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514656:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827588.514657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514660:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514665:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514668:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514671:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827588.514686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332955904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514696:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514699:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827588.514700:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514705:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514711:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514715:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514719:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827588.514729:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332956032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514756:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514768:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827588.514770:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514781:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514788:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514800:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514812:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827588.514826:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332956160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827588.514855:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827588.514859:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827588.514868:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827588.514886:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9399 00000100:00080000:0.0:1787827588.514889:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827588.514894:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827588.514897:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827588.514908:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332956288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827588.898410:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827589.922554:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827590.946431:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827591.970540:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827593.634339:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634353:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827593.634359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634373:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634387:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634396:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827593.634454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332956416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634490:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827593.634498:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634503:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634508:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634513:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827593.634542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332956544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634557:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634564:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827593.634567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634604:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634613:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634630:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634637:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827593.634655:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332956672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634668:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634672:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827593.634675:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634679:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634685:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634689:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634695:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827593.634707:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332956800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634719:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634729:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827593.634736:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634741:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634747:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634752:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634759:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827593.634774:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332956928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634785:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634789:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827593.634791:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634797:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634801:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634834:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634839:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827593.634852:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332957056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634904:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.634913:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827593.634920:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.634933:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.634939:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.634951:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.634961:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827593.634985:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332957184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827593.634998:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827593.635006:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827593.635022:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827593.635030:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9404 00000100:00080000:0.0:1787827593.635037:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827593.635048:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827593.635058:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827593.635084:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332957312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827594.018513:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827595.042398:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827596.066415:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827597.090538:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827598.114380:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827598.754384:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754396:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827598.754401:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754413:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754426:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754434:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827598.754476:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332957440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.754511:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754516:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827598.754520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754526:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754532:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.754567:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827598.754608:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332957568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.754626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827598.754636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754642:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754648:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754655:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.754664:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827598.754687:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332957696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.754704:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827598.754716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754731:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754739:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754750:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.754764:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827598.754784:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332957824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.754825:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754830:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827598.754834:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754846:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754853:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754860:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.754868:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827598.754898:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332957952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.754930:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.754943:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827598.754950:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.754960:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.754985:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.754997:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.755007:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827598.755029:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332958080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.755073:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.755087:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827598.755090:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.755107:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.755117:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.755130:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.755142:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827598.755166:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332958208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827598.755194:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.755203:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827598.755210:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827598.755221:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9409 00000100:00080000:1.0:1787827598.755226:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827598.755233:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827598.755258:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827598.755325:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332958336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827599.138459:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827600.162461:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827601.186422:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827602.210437:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827603.874333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874341:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827603.874344:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874353:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874362:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874367:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827603.874402:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663332958464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874429:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827603.874430:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874435:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874438:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827603.874455:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332958592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827603.874470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874477:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874480:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874482:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827603.874503:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332958720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827603.874530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874537:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874540:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874544:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827603.874558:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332958848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874568:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827603.874569:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874574:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874576:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827603.874593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332958976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874618:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874620:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827603.874622:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874628:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874633:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874639:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827603.874656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332959104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874674:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827603.874678:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874682:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874684:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874686:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874689:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827603.874699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332959232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.874708:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827603.874711:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827603.874712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827603.874714:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9414 00000100:00080000:1.0:1787827603.874717:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827603.874722:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827603.874731:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827603.874744:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332959360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827604.258410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827605.282358:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827606.306563:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827607.330459:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827608.354497:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827608.994405:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994430:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827608.994435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994448:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994462:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.994471:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827608.994595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332959488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.994634:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827608.994651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994657:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994662:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.994667:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.994679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827608.994714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332959616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.994729:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994734:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827608.994739:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994748:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994760:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.994764:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.994771:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827608.994788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332959744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.994809:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994813:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827608.994815:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994820:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994824:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.994838:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.994845:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827608.994867:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332959872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.994879:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994883:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827608.994886:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994891:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994897:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.994902:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.994911:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827608.994938:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332960000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.994953:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.994964:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827608.994971:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.994987:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.994998:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.995014:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.995024:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827608.995043:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332960128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.995061:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.995068:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827608.995076:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.995089:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.995092:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.995098:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.995103:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827608.995142:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332960256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827608.995164:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827608.995180:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827608.995189:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.995199:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9419 00000100:00080000:1.0:1787827608.995207:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.995225:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827608.995242:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827608.995352:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332960384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827609.378441:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827610.402548:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827611.426581:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827612.450674:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827614.114573:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332960512:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827614.114646:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.114694:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de1f80 x1874663332960512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.114719:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332960512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.114743:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.114752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.114761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332960512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827614.114799:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827614.114847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.114872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de1c00 x1874663332960640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.114898:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332960640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.114922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.114927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.114930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827614.114940:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827614.114960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.114979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de1500 x1874663332960768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.114988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332960768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.114995:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.114997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115000:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827614.115004:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827614.115018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.115025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de0a80 x1874663332960896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.115034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332960896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.115040:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.115042:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115044:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332960896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827614.115048:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332961024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827614.115061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.115067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de0000 x1874663332961024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.115077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332961024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.115083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.115085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115087:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332961024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827614.115091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827614.115103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.115110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de3800 x1874663332961152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.115118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332961152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.115132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.115134:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115137:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827614.115151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827614.115164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.115172:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de0e00 x1874663332961280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.115180:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332961280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.115191:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.115194:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827614.115201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827614.115212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827614.115250:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9057d0de3100 x1874663332961408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827614.115266:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332961408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827614.115278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827614.115282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827614.115287:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332961408:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827614.498434:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827615.522414:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827616.546500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827617.570397:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827618.594439:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827619.234454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827619.234473:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234485:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234510:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234520:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827619.234583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663332961536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234624:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827619.234627:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234638:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234643:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234654:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827619.234702:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332961664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234724:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827619.234726:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234736:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234741:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234748:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827619.234762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332961792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234776:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234781:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827619.234784:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234789:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234794:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234799:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827619.234821:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332961920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234837:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234842:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827619.234845:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234851:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234856:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234861:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234868:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827619.234895:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332962048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234904:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234907:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827619.234909:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234913:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234916:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234919:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234933:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827619.234944:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332962176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.234961:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.234965:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827619.234967:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.234971:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.234974:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.234978:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.234983:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827619.234992:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332962304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827619.235003:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827619.235006:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827619.235008:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827619.235012:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9429 00000100:00080000:1.0:1787827619.235015:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827619.235018:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827619.235023:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827619.235032:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332962432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827619.618419:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827620.642579:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827621.666380:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827622.690492:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827624.354625:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332962560:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827624.354730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.354786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de1880 x1874663332962560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.354830:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332962560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.354874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.354886:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.354895:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332962560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827624.354942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827624.354984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de0700 x1874663332962688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332962688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355069:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827624.355090:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827624.355146:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de2a00 x1874663332962816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355277:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332962816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355317:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355336:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827624.355355:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827624.355420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355438:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de1180 x1874663332962944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355453:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332962944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355466:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355470:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332962944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827624.355491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827624.355524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355535:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de1880 x1874663332963072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355547:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332963072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355575:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355579:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355582:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827624.355588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827624.355606:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355615:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de3100 x1874663332963200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355628:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332963200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355643:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827624.355649:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827624.355666:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355675:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de0e00 x1874663332963328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332963328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355695:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827624.355709:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827624.355726:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827624.355736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827624/real 1787827624] req@ffff9057d0de3800 x1874663332963456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827624.355748:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332963456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827624.355757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827624.355760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827624.355763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332963456:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827624.738409:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827625.762405:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827626.786490:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827627.810536:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827628.834373:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827629.474372:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474381:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827629.474384:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474393:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474402:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474408:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827629.474442:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332963584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474465:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827629.474473:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474476:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474478:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474481:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474489:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827629.474498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332963712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474510:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827629.474512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332963584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827629.474513:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474522:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474524:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474526:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474529:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827629.474538:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332963840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474544:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474546:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827629.474548:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474550:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474553:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474555:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827629.474564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332963968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474574:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827629.474575:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474577:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474579:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474582:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787827629.474586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474590:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827629.474606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332964096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474617:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827629.474623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827629.474628:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158ce00 x1874663332963584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827629.474632:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474640:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474647:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827629.474651:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663332963584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827629.474655:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827629.474665:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.474669:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787827629.474672:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332964224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827629.474676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332963584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827629.474690:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827629.474694:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9440 00000100:00080000:0.0:1787827629.474701:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474705:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474708:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827629.474711:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827629.474719:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332964352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827629.474730:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827629.474734:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827629.474736:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827629.474742:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9440 00000800:00000100:1.0:1787827629.474746:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827629.474747:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827629.474755:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827629.474762:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827629.474762:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158d880 x1874663332963712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.474777:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663332963712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827629.474782:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332964480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827629.474790:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.474798:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.474802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827629.474807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827629.474822:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.474829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158d180 x1874663332963840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.474837:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663332963840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.474842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.474844:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.474846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827629.474858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827629.474871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.474878:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158fb80 x1874663332963968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.474907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663332963968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.474912:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.474914:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.474916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332963968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827629.474921:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827629.474933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.474940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158ed80 x1874663332964096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.474953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663332964096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.474980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.474984:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.474989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827629.474998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827629.475027:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.475057:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158d500 x1874663332964224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.475065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663332964224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.475075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.475077:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.475079:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827629.475094:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827629.475108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.475114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158c700 x1874663332964352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.475139:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663332964352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.475147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.475150:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.475152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827629.475158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827629.475175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827629.475183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827629/real 1787827629] req@ffff9057f158ea00 x1874663332964480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827629.475194:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663332964480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827629.475201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827629.475203:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827629.475207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332964480:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827629.858547:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827630.882508:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827631.906699:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827632.930684:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827634.594338:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594348:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827634.594353:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594363:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594376:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594383:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827634.594434:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332964608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827634.594466:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594469:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827634.594470:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594474:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594476:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594479:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594489:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827634.594503:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332964736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827634.594513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827634.594519:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594522:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594527:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594538:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827634.594551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332964864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827634.594558:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332964608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827634.594570:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827634.594577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594582:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594587:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594592:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827634.594614:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332964992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827634.594622:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827634.594626:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594630:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594633:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594636:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594639:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827634.594645:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787827634.594650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332965120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827634.594658:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594661:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827634.594662:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594665:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594668:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827634.594671:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594674:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827634.594682:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332965248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827634.594692:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594695:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827634.594696:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827634.594699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594702:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827634.594706:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158f100 x1874663332964608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827634.594708:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827634.594714:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827634.594727:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332965376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827634.594734:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663332964608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827634.594736:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827634.594746:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827634.594750:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827634.594756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827634.594757:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9445 00000100:00080000:0.0:1787827634.594764:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827634.594765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827634.594772:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827634.594773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332964608:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787827634.594779:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827634.594783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827634.594790:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332965504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827634.594807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.594818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158e300 x1874663332964736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.594827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663332964736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.594833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.594835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.594838:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.594862:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827634.594873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.594880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158dc00 x1874663332964864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.594887:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663332964864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.594892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.594894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.594896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.594899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827634.594910:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.594915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158c000 x1874663332964992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.594922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663332964992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.594927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.594929:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.594931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332964992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827634.594934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827634.594944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.594950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158df80 x1874663332965120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.594958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663332965120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.594963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.594965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.594984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827634.594988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827634.595001:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.595008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158e680 x1874663332965248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.595016:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663332965248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.595022:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.595024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.595026:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.595030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827634.595041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.595047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158f800 x1874663332965376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.595054:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663332965376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.595065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.595067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.595069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.595073:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827634.595084:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827634.595090:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff9057f158ca80 x1874663332965504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.595098:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663332965504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.595103:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.595106:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827634.595108:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332965504:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827634.978753:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827636.002415:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827637.026541:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827638.050435:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827639.074434:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827639.714292:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714300:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827639.714304:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827639.714312:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714321:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827639.714327:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827639.714359:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332965632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714380:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714383:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827639.714384:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827639.714387:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714390:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827639.714392:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714398:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827639.714408:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332965760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714416:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714419:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827639.714420:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827639.714422:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714426:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827639.714427:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332965632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827639.714431:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714437:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827639.714446:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332965888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714458:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714461:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827639.714462:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827639.714465:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714475:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827639.714478:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827639.714486:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787827639.714491:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332966016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714499:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714502:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827639.714513:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827639.714516:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714518:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827639.714520:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714524:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827639.714533:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158ca80 x1874663332965632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827639.714534:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332966144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714561:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714564:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827639.714565:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827639.714579:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663332965632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827639.714580:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714586:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827639.714592:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827639.714604:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332966272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714614:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714617:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827639.714617:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827639.714619:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827639.714628:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827639.714629:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714644:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827639.714647:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332965632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827639.714650:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714659:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827639.714665:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827639.714668:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332966400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827639.714682:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827639.714693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827639.714696:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827639.714700:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827639.714715:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9450 00000100:00080000:0.0:1787827639.714721:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827639.714725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158f800 x1874663332965760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827639.714729:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827639.714738:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827639.714744:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663332965760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827639.714752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332966528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827639.714754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.714762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.714779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.714807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827639.714823:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.714841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158e680 x1874663332965888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.714849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663332965888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.714854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.714856:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.714858:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332965888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.714862:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827639.714873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.714879:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158df80 x1874663332966016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.714886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663332966016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.714891:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.714893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.714894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827639.714904:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827639.714913:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.714919:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158c000 x1874663332966144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.714926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663332966144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.714931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.714932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.714934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827639.714937:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827639.714947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.714952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158dc00 x1874663332966272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.714963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663332966272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.714968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.714970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.714972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.714975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827639.714984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.714989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158e300 x1874663332966400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.714996:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663332966400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.715001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.715002:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.715004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.715007:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827639.715017:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827639.715023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff9057f158f100 x1874663332966528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.715029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663332966528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.715049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.715053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827639.715056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332966528:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827640.098526:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827641.122419:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827642.146409:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827643.170542:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827644.834486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332966656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827644.834629:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.834699:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.834753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332967040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827644.834790:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332967168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827644.834804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.834822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.834835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967552:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827645.218372:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827646.242413:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827647.266422:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827647.906637:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de1500 x1874663332966912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906671:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332966912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906694:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.906714:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de1c00 x1874663332967040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332967040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906738:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906742:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906746:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332967040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827647.906754:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de1f80 x1874663332967168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906768:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332967168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906778:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906782:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332967168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827647.906794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de3800 x1874663332967296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906806:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332967296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906814:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906821:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.906827:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de0000 x1874663332966656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332966656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906849:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906853:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332966656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827647.906864:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de0a80 x1874663332966784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332966784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906889:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906893:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332966784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.906940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de0e00 x1874663332967424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332967424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.906965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.906970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.906974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.906981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827647] req@ffff9057d0de3100 x1874663332967552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.906994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332967552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.907003:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.907007:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827647.907011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332967552:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827648.290353:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827649.314474:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827649.954370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954381:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827649.954386:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954395:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954407:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954414:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827649.954448:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332967680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954484:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827649.954486:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954492:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954501:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954511:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827649.954527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332967808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827649.954568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954582:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954587:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827649.954610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332967936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954621:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954625:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827649.954628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954633:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827649.954672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332968064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827649.954706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954714:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954721:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954727:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954737:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827649.954763:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332968192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954801:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954810:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827649.954824:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954840:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954849:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954856:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954862:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827649.954881:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332968320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.954894:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.954914:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827649.954919:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.954927:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.954936:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.954952:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.954965:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827649.954989:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332968448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827649.955012:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827649.955023:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827649.955027:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827649.955049:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9460 00000100:00080000:1.0:1787827649.955060:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827649.955081:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827649.955090:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827649.955119:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332968576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827650.338398:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827651.362373:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827652.386502:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827653.410476:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827655.074327:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074335:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827655.074338:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074347:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074355:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074361:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827655.074395:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663332968704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074425:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074428:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.074429:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074432:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074435:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074437:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074444:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.074454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332968832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074461:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074463:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.074464:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074468:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.074478:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332968960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.074494:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074497:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074499:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074502:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.074515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332969088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074524:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074526:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.074527:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074529:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074532:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074534:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074537:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.074546:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332969216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074552:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074555:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.074556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074559:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074561:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.074574:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332969344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074590:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074593:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.074595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074598:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074602:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074604:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.074617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332969472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.074626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.074630:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.074635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.074645:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9465 00000100:00080000:1.0:1787827655.074653:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.074660:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827655.074677:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.074694:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332969600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827655.458369:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827656.482398:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827657.506327:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827659.554368:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827660.194272:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194280:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827660.194283:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194291:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194300:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194306:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827660.194337:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332969728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194360:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194362:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.194363:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194366:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194368:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194371:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194378:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827660.194386:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332969856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194394:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194398:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.194400:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194408:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194411:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194413:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194416:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827660.194428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332969984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.194444:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194449:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194453:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194459:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827660.194485:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332970112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194493:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.194497:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194500:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194503:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194509:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827660.194531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332970240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.194572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194576:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194579:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194583:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194588:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827660.194602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332970368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.194619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827660.194648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332970496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.194667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.194671:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.194674:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.194682:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9470 00000100:00080000:1.0:1787827660.194686:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.194694:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827660.194700:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827660.194712:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332970624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827660.578423:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827661.602356:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827662.626389:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827663.650457:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827665.314339:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314350:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827665.314354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314365:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314386:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827665.314424:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663332970752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827665.314457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314462:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314480:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827665.314499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332970880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827665.314519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314524:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314527:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314532:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314538:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827665.314554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332971008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314567:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827665.314573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314578:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314583:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314587:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314593:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827665.314609:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332971136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827665.314652:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314663:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314668:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314672:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827665.314698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332971264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314727:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827665.314731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314740:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314745:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314756:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314764:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827665.314783:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332971392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314803:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827665.314810:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314829:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314840:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314850:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314859:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827665.314881:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332971520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827665.314901:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827665.314914:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827665.314916:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827665.314926:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9476 00000100:00080000:1.0:1787827665.314936:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827665.314943:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827665.314953:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827665.314972:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332971648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827665.698365:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827666.722426:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827667.746390:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827669.794392:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827670.434496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332971776:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827670.434567:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.434625:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de1880 x1874663332971776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.434658:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332971776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.434688:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.434700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.434710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332971776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827670.434754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332971904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827670.434794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.434814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de1180 x1874663332971904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.434832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332971904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.434842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.434846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.434850:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332971904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827670.434857:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827670.434876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.434887:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de2a00 x1874663332972032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.434900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332972032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.434909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.434912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.434916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827670.434922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827670.434940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.434950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de0700 x1874663332972160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.434962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332972160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.434971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.434974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.434978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827670.434985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827670.435003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.435014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de3100 x1874663332972288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.435033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332972288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.435042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.435061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.435065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827670.435072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827670.435092:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.435103:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de0e00 x1874663332972416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.435115:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332972416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.435124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.435127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.435131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827670.435151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827670.435172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.435184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de0a80 x1874663332972544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.435196:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332972544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.435211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.435215:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.435218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827670.435247:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827670.435267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827670.435278:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827670/real 1787827670] req@ffff9057d0de0000 x1874663332972672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827670.435291:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332972672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827670.435300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827670.435303:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827670.435306:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332972672:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827670.818512:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827671.842417:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827672.866373:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827673.890474:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827675.554316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827675.554325:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827675.554329:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827675.554337:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554347:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554361:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827675.554395:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332972800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554421:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827675.554425:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827675.554427:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827675.554433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554437:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554453:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827675.554470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332972928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827675.554487:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332972800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827675.554488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827675.554491:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827675.554500:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554503:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554507:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554511:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827675.554522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332973056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827675.554533:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827675.554534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827675.554536:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554540:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554543:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554547:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827675.554555:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787827675.554557:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332973184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554565:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827675.554569:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827675.554571:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827675.554575:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554578:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554580:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554583:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827675.554593:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332973312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554599:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827675.554602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827675.554605:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827675.554607:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158ea00 x1874663332972800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827675.554612:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554618:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554632:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827675.554643:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332973440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827675.554647:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663332972800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827675.554676:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827675.554682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827675.554683:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827675.554689:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827675.554691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827675.554702:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00100000:1.0:1787827675.554703:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332972800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827675.554708:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827675.554714:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827675.554729:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827675.554736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332973568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827675.554751:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827675.554753:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827675.554760:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827675.554763:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827675.554768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158c700 x1874663332972928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827675.554770:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9486 00000100:00080000:0.0:1787827675.554779:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827675.554786:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663332972928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827675.554788:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827675.554797:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827675.554799:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.554805:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.554807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332972928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787827675.554812:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332973696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827675.554816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827675.554841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.554852:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158d500 x1874663332973056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.554863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663332973056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.554869:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.554872:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.554874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827675.554879:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827675.554891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.554897:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158ed80 x1874663332973184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.554905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663332973184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.554911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.554912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.554914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827675.554918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827675.554929:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.554934:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158fb80 x1874663332973312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.554942:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663332973312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.554948:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.554950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.554952:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827675.554955:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827675.554966:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.554972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158d180 x1874663332973440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.554981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663332973440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.554986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.554988:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.554990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827675.555003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827675.555037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.555047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158d880 x1874663332973568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.555055:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663332973568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.555061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.555063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.555066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827675.555070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827675.555082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827675.555089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827675/real 1787827675] req@ffff9057f158ce00 x1874663332973696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827675.555097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663332973696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827675.555108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827675.555110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827675.555112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332973696:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827675.938485:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827676.962386:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827677.986515:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827680.034373:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827680.674316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674326:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827680.674330:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827680.674340:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674350:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674358:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827680.674396:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663332973824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827680.674443:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674447:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827680.674450:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827680.674454:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827680.674488:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663332973952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827680.674492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332973824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827680.674518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827680.674524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827680.674529:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674533:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674543:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827680.674556:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663332974080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827680.674577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674581:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827680.674585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827680.674586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827680.674593:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674599:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674604:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674611:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827680.674623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663332974208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827680.674643:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674648:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827680.674651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827680.674655:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158ce00 x1874663332973824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827680.674657:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674664:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674671:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674678:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827680.674679:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663332973824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.674691:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827680.674693:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663332974336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827680.674697:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.674706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332973824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787827680.674712:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674717:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827680.674719:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827680.674725:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674729:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674744:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827680.674749:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973952:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787827680.674751:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827680.674769:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663332974464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827680.674781:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827680.674788:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787827680.674788:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787827680.674793:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827680.674799:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674804:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827680.674809:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158d880 x1874663332973952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827680.674812:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674832:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827680.674842:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663332973952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827680.674847:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663332974592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827680.674864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.674871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787827680.674872:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827680.674876:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332973952:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787827680.674879:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827680.674884:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827680.674886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974080:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787827680.674892:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9491 00000100:00080000:0.0:1787827680.674897:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827680.674902:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827680.674908:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827680.674909:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.674922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158d180 x1874663332974080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827680.674926:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663332974720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827680.674954:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663332974080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.674966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.674970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.674990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827680.674998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827680.675020:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.675037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158fb80 x1874663332974208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827680.675046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663332974208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.675051:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.675054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.675056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827680.675060:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827680.675072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.675079:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158ed80 x1874663332974336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827680.675086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663332974336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.675091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.675093:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.675095:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332974336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827680.675108:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827680.675119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.675125:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158d500 x1874663332974464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827680.675132:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663332974464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.675137:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.675139:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.675141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827680.675144:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827680.675154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.675159:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158c700 x1874663332974592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827680.675166:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663332974592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.675171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.675173:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.675175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827680.675178:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827680.675187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827680.675192:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff9057f158ea00 x1874663332974720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827680.675216:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663332974720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827680.675221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827680.675224:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827680.675226:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332974720:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827681.058395:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827682.082382:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827683.106368:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827684.130433:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827685.794395:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794403:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827685.794406:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794414:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794424:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794429:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827685.794480:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332974848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794524:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794527:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827685.794529:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794533:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794540:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827685.794576:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332974976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794585:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827685.794591:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794596:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794603:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794606:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827685.794624:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332975104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827685.794635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794638:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794641:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794644:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794648:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827685.794657:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332975232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794665:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794667:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827685.794669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794672:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794674:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794677:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794679:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827685.794686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332975360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794692:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794695:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827685.794696:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794701:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794718:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794724:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827685.794735:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332975488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794746:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794749:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827685.794752:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794756:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794761:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794767:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794773:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827685.794784:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332975616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827685.794807:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827685.794810:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827685.794811:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827685.794814:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9496 00000100:00080000:1.0:1787827685.794816:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827685.794818:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827685.794822:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827685.794833:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332975744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827686.178393:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827687.202370:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827688.226385:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827690.274377:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827690.914499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332975872:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827690.914574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de3800 x1874663332975872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914648:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332975872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914677:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914689:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332975872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827690.914720:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827690.914752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de1f80 x1874663332976000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914792:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332976000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914805:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827690.914813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827690.914830:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de1c00 x1874663332976128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332976128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827690.914865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827690.914876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de1500 x1874663332976256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914890:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332976256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914897:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827690.914902:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827690.914931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de0000 x1874663332976384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332976384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914955:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332976384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827690.914962:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827690.914974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.914980:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de0a80 x1874663332976512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.914988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332976512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.914993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.914994:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.914996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827690.915000:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827690.915010:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.915015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de0e00 x1874663332976640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.915023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332976640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.915028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.915029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.915031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827690.915046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827690.915056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827690.915063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff9057d0de3100 x1874663332976768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827690.915071:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332976768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827690.915076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827690.915078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827690.915080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332976768:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827691.298420:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827692.322490:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827693.346505:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827694.370508:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827696.034326:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827696.034341:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034352:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034365:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827696.034412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663332976896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034447:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827696.034454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034482:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827696.034503:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332977024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034519:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827696.034526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034530:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034536:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034542:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827696.034568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332977152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827696.034589:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034600:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034610:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034617:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827696.034650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332977280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827696.034679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034689:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034696:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034702:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827696.034731:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332977408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034756:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034766:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827696.034773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034784:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034794:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034801:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034812:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827696.034833:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332977536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034856:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827696.034861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034890:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.034900:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.034927:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.034937:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827696.034956:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332977664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827696.034974:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.034984:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827696.034988:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.034997:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9506 00000100:00080000:1.0:1787827696.035005:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827696.035013:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827696.035024:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827696.035044:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332977792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827696.418398:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827697.442434:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827698.466495:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827700.514374:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827701.154373:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154387:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827701.154391:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154403:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154414:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154422:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827701.154474:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332977920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154509:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154513:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827701.154515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154519:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154522:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154526:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154535:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827701.154548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332978048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154561:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154565:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827701.154567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154570:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154573:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154577:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154581:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827701.154591:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332978176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154611:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827701.154633:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154639:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154642:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154645:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154651:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827701.154670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332978304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154684:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154690:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827701.154692:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154698:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154707:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154714:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154734:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827701.154754:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332978432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154786:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154790:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827701.154794:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154804:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154815:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154821:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154828:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827701.154868:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332978560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154880:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154884:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827701.154886:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154901:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154906:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.154916:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.154924:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827701.154950:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332978688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827701.154977:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827701.154982:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827701.154984:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827701.154990:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9511 00000100:00080000:0.0:1787827701.154994:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827701.155005:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827701.155021:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827701.155043:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332978816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827701.538413:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827702.562488:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827703.586462:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827704.610487:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827706.274350:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274359:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827706.274363:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274381:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274390:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274396:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827706.274446:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332978944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274474:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827706.274477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274480:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274483:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274486:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274493:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827706.274501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332979072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274508:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274511:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827706.274512:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274540:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274544:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274547:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827706.274558:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332979200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827706.274603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274618:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274622:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274625:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274629:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827706.274653:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332979328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274664:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827706.274666:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274668:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274670:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274673:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827706.274684:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332979456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274691:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827706.274695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274698:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274701:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274704:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274708:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827706.274715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332979584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274729:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274733:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827706.274734:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274739:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274742:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274746:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274752:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827706.274759:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332979712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827706.274767:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827706.274770:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827706.274772:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827706.274777:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9517 00000100:00080000:1.0:1787827706.274779:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827706.274781:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827706.274784:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827706.274790:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332979840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827706.658415:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827707.682428:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827708.706454:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827710.754421:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827711.394316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394326:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827711.394330:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394340:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394359:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827711.394397:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663332979968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394425:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394430:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827711.394432:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394436:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394441:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394447:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394456:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827711.394471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663332980096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394487:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827711.394501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394508:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394512:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394517:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394523:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827711.394555:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663332980224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394568:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827711.394576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394581:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394586:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394590:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394597:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827711.394611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663332980352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394624:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827711.394642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394649:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394653:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394660:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827711.394697:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663332980480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394713:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394721:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827711.394724:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394731:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394737:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394750:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394760:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827711.394818:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663332980608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394845:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394849:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827711.394862:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394869:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394874:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394888:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394896:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827711.394912:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663332980736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827711.394937:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827711.394942:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827711.394944:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827711.394952:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9522 00000100:00080000:0.0:1787827711.394957:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827711.394968:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827711.394975:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827711.394993:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663332980864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827711.778395:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827712.802421:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827713.826638:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827714.466625:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827714.466687:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827714.466712:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466798:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466808:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466815:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466827:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466844:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466854:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827714.466864:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:1.0:1787827714.850529:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827716.514469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332980992:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827716.514547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.514594:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de0700 x1874663332980992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.514622:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332980992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.514643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.514653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.514663:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332980992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827716.514694:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827716.514741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.514765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de2a00 x1874663332981120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.514787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332981120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.514805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.514813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.514823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827716.514838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827716.514871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.514888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de1180 x1874663332981248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.514905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332981248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.514914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.514918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.514922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827716.514929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827716.514951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.514964:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de1880 x1874663332981376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.514975:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332981376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.514983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.514987:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.514990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827716.514996:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827716.515022:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.515036:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de3100 x1874663332981504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.515049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332981504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.515057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.515061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.515064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332981504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827716.515082:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827716.515105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.515117:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de0e00 x1874663332981632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.515131:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332981632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.515140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.515149:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.515152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827716.515158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827716.515179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.515191:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de0a80 x1874663332981760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.515203:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332981760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.515210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.515213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.515216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827716.515267:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827716.515290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827716.515302:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827716/real 1787827716] req@ffff9057d0de0000 x1874663332981888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827716.515315:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332981888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827716.515329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827716.515332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827716.515335:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332981888:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827716.898400:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827717.922357:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827718.946425:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827720.994437:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827721.634474:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332982016:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827721.634568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de1500 x1874663332982016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332982016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634687:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634715:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332982016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827721.634730:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827721.634757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634771:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de1c00 x1874663332982144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332982144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827721.634811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827721.634824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634831:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de1f80 x1874663332982272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332982272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827721.634853:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827721.634864:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de3800 x1874663332982400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634880:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332982400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634885:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827721.634892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827721.634903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634918:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de0000 x1874663332982528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332982528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332982528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827721.634939:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827721.634949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de0a80 x1874663332982656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.634962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332982656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.634967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.634969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.634971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827721.634980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827721.634991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.634997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de0e00 x1874663332982784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.635004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332982784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.635009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.635012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.635014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827721.635017:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827721.635027:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827721.635033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff9057d0de3100 x1874663332982912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827721.635040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332982912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827721.635045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827721.635048:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827721.635050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332982912:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827722.018447:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827723.042447:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827724.066471:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827725.090513:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827726.754747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332983040:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827726.754878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.754943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de1880 x1874663332983040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.754980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332983040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332983040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827726.755062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827726.755119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755154:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de1180 x1874663332983168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755195:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332983168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755270:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755282:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827726.755301:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827726.755355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de2a00 x1874663332983296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755419:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332983296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755449:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755458:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827726.755468:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827726.755496:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de0700 x1874663332983424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755566:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332983424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755581:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827726.755608:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827726.755622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755628:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de3100 x1874663332983552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332983552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755643:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755645:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332983552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827726.755650:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827726.755680:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755688:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de0e00 x1874663332983680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755701:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332983680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755707:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755710:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755712:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827726.755716:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827726.755727:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755733:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de0a80 x1874663332983808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755741:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332983808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755746:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755750:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827726.755754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827726.755764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827726.755771:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff9057d0de0000 x1874663332983936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827726.755777:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332983936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827726.755787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827726.755789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827726.755791:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332983936:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827727.138379:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827728.162527:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827729.186350:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827731.234455:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827731.874464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827731.874504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874512:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874521:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874527:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827731.874602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663332984064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874679:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827731.874685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874691:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874696:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874702:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827731.874732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663332984192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874748:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874752:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.874754:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874757:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874760:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874765:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874771:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827731.874782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663332984320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874791:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827731.874793:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874797:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874800:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874803:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827731.874814:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663332984448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874819:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874822:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.874823:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874826:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874828:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874830:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874833:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827731.874847:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663332984576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874855:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874857:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827731.874859:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874862:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874876:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874878:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874883:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827731.874889:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663332984704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874894:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874897:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.874898:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874901:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874905:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874908:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874915:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827731.874924:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663332984832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.874934:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.874947:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.874952:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.874968:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9542 00000100:00080000:1.0:1787827731.874979:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.874984:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827731.874989:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827731.875003:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663332984960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827732.258425:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827733.282374:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827734.306547:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827735.330660:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827736.994329:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827736.994345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994356:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827736.994405:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663332985088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994427:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827736.994443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994460:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827736.994473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332985216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994484:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827736.994492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994495:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994503:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827736.994515:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332985344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827736.994526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994528:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994530:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994532:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827736.994546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332985472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827736.994557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994564:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994567:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827736.994579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332985600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994605:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827736.994621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994640:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994648:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827736.994672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332985728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994693:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827736.994698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994702:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994706:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994708:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994711:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827736.994720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332985856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827736.994731:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827736.994734:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827736.994735:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827736.994738:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9547 00000100:00080000:1.0:1787827736.994741:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827736.994743:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827736.994747:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827736.994758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332985984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827737.378467:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827738.402391:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827739.426385:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827741.474344:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827742.114500:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332986112:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827742.114582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.114629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de3800 x1874663332986112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.114664:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332986112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.114682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.114692:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.114702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332986112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827742.114718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827742.114753:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.114769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de1f80 x1874663332986240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.114782:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332986240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.114792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.114796:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.114800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827742.114822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827742.114844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.114856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de1c00 x1874663332986368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.114869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332986368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.114877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.114880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.114884:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827742.114890:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827742.114910:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.114922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de1500 x1874663332986496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.114934:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332986496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.114943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.114947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.114951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827742.114957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827742.114977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.114989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de0000 x1874663332986624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.115001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332986624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.115009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.115013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.115016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332986624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827742.115022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332986752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827742.115041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.115052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de0a80 x1874663332986752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.115064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332986752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.115073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.115076:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.115080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332986752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827742.115086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332986880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827742.115105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.115117:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de0e00 x1874663332986880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.115128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332986880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.115136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.115139:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.115143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332986880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827742.115149:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827742.115169:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827742.115179:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9057d0de3100 x1874663332987008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827742.115191:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332987008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827742.115198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827742.115201:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827742.115205:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987008:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827742.498420:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827743.522394:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827744.546412:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827745.570399:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827747.234599:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332987136:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827747.234705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.234792:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de0700 x1874663332987136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.234817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332987136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.234852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.234858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.234864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332987136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827747.234878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827747.234916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.234951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de2a00 x1874663332987264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.234965:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332987264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.234975:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.234979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.234983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827747.234991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827747.235012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de1180 x1874663332987392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235036:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332987392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827747.235071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827747.235085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de1880 x1874663332987520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235101:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332987520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235110:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827747.235115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827747.235126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235132:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de3100 x1874663332987648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332987648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235153:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332987648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827747.235159:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827747.235170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235176:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de0e00 x1874663332987776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235184:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332987776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235194:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827747.235198:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827747.235209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235215:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de0a80 x1874663332987904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235222:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332987904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235229:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235232:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332987904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827747.235236:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332988032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827747.235248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827747.235297:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827747/real 1787827747] req@ffff9057d0de0000 x1874663332988032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827747.235311:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332988032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827747.235320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827747.235325:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827747.235329:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332988032:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827747.618397:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827748.642582:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827749.666422:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827751.714541:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827752.354364:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354374:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827752.354378:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354389:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354401:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354408:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827752.354444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663332988160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354486:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827752.354499:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354505:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354510:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354515:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827752.354545:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663332988288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354569:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827752.354576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354580:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354584:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354591:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827752.354613:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663332988416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354624:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354627:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827752.354629:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354636:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354641:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354648:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827752.354671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663332988544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827752.354719:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354728:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354732:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354737:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354745:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827752.354761:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663332988672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354804:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827752.354808:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354817:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354824:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354830:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354839:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827752.354859:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663332988800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354902:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354909:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827752.354913:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.354921:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.354928:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.354945:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.354955:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827752.354971:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663332988928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827752.354990:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827752.354996:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827752.355000:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827752.355016:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9563 00000100:00080000:1.0:1787827752.355021:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827752.355028:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827752.355035:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827752.355052:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663332989056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827752.738351:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827753.762460:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827754.786363:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827755.810508:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827757.474529:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332989184:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827757.474607:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.474668:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de1500 x1874663332989184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.474717:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332989184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.474747:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.474758:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.474767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332989184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827757.474794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827757.474820:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.474834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de1c00 x1874663332989312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.474848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332989312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.474857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.474860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.474864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827757.474871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827757.474891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.474903:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de1f80 x1874663332989440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.474917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332989440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.474926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.474930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.474934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827757.474940:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827757.474961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.474973:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de3800 x1874663332989568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.474987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332989568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.474996:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.475000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.475003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827757.475010:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827757.475031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.475044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de0000 x1874663332989696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.475057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332989696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.475066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.475070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.475073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332989696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827757.475079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332989824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827757.475100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.475112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de0a80 x1874663332989824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.475133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332989824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.475142:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.475146:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.475150:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332989824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827757.475157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332989952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827757.475191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.475204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de0e00 x1874663332989952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.475218:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332989952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.475250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.475254:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.475257:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332989952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827757.475264:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827757.475285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827757.475296:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827757/real 1787827757] req@ffff9057d0de3100 x1874663332990080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827757.475325:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332990080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827757.475342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827757.475345:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827757.475349:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990080:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827757.858392:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827758.882376:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827759.906398:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827761.954567:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827762.594508:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332990208:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827762.594572:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594616:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de1880 x1874663332990208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594651:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332990208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594681:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594692:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332990208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827762.594725:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827762.594752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de1180 x1874663332990336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332990336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594796:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827762.594807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827762.594828:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594840:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de2a00 x1874663332990464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332990464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827762.594876:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827762.594891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de0700 x1874663332990592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332990592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827762.594917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827762.594927:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594933:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de3100 x1874663332990720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594940:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332990720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332990720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827762.594958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827762.594968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.594975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de0e00 x1874663332990848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.594982:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332990848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.594986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.594988:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.594990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827762.594993:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827762.595002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.595008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de0a80 x1874663332990976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.595015:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332990976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.595020:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.595022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.595024:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332990976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827762.595037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332991104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827762.595047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827762.595052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff9057d0de0000 x1874663332991104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827762.595059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332991104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827762.595065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827762.595066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827762.595068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332991104:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827762.978397:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827764.002513:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827765.026372:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827766.050497:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827767.714559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332991232:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827767.714643:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.714698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de3800 x1874663332991232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.714736:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332991232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.714755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.714765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.714771:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332991232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827767.714779:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827767.714798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.714809:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de1f80 x1874663332991360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.714818:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332991360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.714823:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.714825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.714828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827767.714853:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827767.714874:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.714885:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de1c00 x1874663332991488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.714897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332991488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.714920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.714924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.714928:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827767.714934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827767.714958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.714970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de1500 x1874663332991616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.714981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332991616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.714989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.714992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.714996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827767.715002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827767.715023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.715031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de0000 x1874663332991744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.715039:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332991744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.715045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.715047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.715049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332991744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827767.715053:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332991872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827767.715065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.715071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de0a80 x1874663332991872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.715083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332991872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.715088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.715090:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.715091:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332991872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827767.715095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827767.715105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.715111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de0e00 x1874663332992000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.715118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332992000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.715123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.715124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.715126:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827767.715129:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827767.715143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827767.715154:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff9057d0de3100 x1874663332992128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827767.715166:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332992128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827767.715180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827767.715183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827767.715187:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992128:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827768.098403:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827769.122454:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827770.147339:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827772.194395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827772.834445:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332992256:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827772.834507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834546:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de0700 x1874663332992256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834568:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332992256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834581:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834589:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834596:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332992256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827772.834619:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827772.834649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de2a00 x1874663332992384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834689:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332992384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834698:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834702:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827772.834713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827772.834736:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de1180 x1874663332992512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834754:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332992512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834759:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827772.834767:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827772.834780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de1880 x1874663332992640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834793:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332992640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834800:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827772.834805:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827772.834814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de3100 x1874663332992768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332992768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332992768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827772.834839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827772.834849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de0e00 x1874663332992896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332992896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332992896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827772.834899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827772.834917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834925:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de0a80 x1874663332993024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332993024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827772.834946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827772.834956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827772.834962:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff9057d0de0000 x1874663332993152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827772.834969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332993152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827772.834974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827772.834976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827772.834977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993152:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827773.218484:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827774.242456:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827775.266413:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827776.290463:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827777.954566:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332993280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827777.954703:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827777.954753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827777.954805:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827777.954884:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827777.954967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827777.955028:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827777.955182:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994176:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827778.338520:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827779.362487:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827780.386428:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827781.026527:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827781.026570:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827781.026589:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026624:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026633:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026644:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026655:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026666:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827781.026674:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de3800 x1874663332993664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827781.026677:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827781.026690:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827781.026714:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332993664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026728:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026738:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827781.026747:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de1c00 x1874663332993408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026760:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332993408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026768:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026772:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026776:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827781.026783:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de1f80 x1874663332993536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026801:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332993536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026816:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827781.026823:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de1500 x1874663332993280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332993280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332993280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827781.026875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de3100 x1874663332994176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026887:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332994176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026898:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827781.026908:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de0e00 x1874663332994048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332994048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026931:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827781.026941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de0a80 x1874663332993920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332993920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026964:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.026967:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332993920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827781.026974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827781] req@ffff9057d0de0000 x1874663332993792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827781.026986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332993792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827781.026993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827781.026997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827781.027000:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332993792:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827782.434404:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827783.074498:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332994304:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827783.074569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.074614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de1880 x1874663332994304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.074648:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332994304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.074678:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.074686:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.074695:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332994304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827783.074749:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827783.074791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.074815:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de1180 x1874663332994432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.074858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332994432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.074884:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.074894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.074906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827783.074923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827783.074968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.074991:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de2a00 x1874663332994560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332994560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827783.075083:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827783.075115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.075132:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de0700 x1874663332994688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075151:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332994688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827783.075193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827783.075258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.075294:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de0000 x1874663332994816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075311:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332994816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075341:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075345:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332994816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827783.075376:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827783.075407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.075430:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de0a80 x1874663332994944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075452:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332994944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075462:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075471:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075475:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332994944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827783.075483:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827783.075511:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.075525:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de0e00 x1874663332995072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075541:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332995072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075558:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075563:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827783.075588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827783.075622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827783.075645:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9057d0de3100 x1874663332995200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827783.075662:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332995200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.075674:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.075678:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827783.075683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995200:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827783.458448:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827784.482503:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827785.506519:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827786.530443:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827788.194506:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332995328:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827788.194576:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194613:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de1500 x1874663332995328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332995328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194665:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.194671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.194676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332995328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827788.194685:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827788.194713:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194744:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de1f80 x1874663332995456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194774:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332995456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.194790:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.194794:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827788.194818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827788.194840:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de1c00 x1874663332995584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332995584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.194866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.194869:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827788.194873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827788.194885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de3800 x1874663332995712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194903:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332995712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.194910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.194912:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827788.194915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827788.194925:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de3100 x1874663332995840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194938:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332995840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.194945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.194947:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332995840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827788.194950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827788.194960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.194966:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de0e00 x1874663332995968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.194973:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332995968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.194998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.195000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.195002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332995968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827788.195005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827788.195015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.195021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de0a80 x1874663332996096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.195027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332996096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.195032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.195033:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.195035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827788.195038:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827788.195064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827788.195070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9057d0de0000 x1874663332996224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.195077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332996224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.195082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.195084:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827788.195086:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996224:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827788.578378:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827789.602384:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827790.626454:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827792.674439:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827793.314464:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332996352:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827793.315665:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315700:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de0700 x1874663332996352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315714:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332996352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315724:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315727:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332996352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827793.315746:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827793.315763:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315772:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de2a00 x1874663332996480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332996480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315791:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827793.315795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827793.315808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de1180 x1874663332996608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332996608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315826:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827793.315833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827793.315844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de1880 x1874663332996736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332996736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315861:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315863:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827793.315868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827793.315878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de0000 x1874663332996864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332996864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315904:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332996864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827793.315907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827793.315917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de0a80 x1874663332996992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332996992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332996992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827793.315941:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332997120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827793.315951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315957:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de0e00 x1874663332997120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.315963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332997120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.315973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.315975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.315977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332997120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827793.315980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332997248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827793.315990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827793.315995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827793/real 1787827793] req@ffff9057d0de3100 x1874663332997248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827793.316002:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332997248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827793.316007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827793.316009:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827793.316011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332997248:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827793.698385:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827794.722517:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827795.746382:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827796.770531:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827798.434465:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332997376:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827798.434571:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de3800 x1874663332997376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434671:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663332997376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.434720:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.434727:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332997376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827798.434773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827798.434802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de1c00 x1874663332997504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434822:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663332997504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434829:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.434831:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.434834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827798.434839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827798.434851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434858:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de1f80 x1874663332997632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663332997632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.434873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.434875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827798.434878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827798.434891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de1500 x1874663332997760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663332997760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.434913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.434915:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827798.434918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827798.434930:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434936:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de3100 x1874663332997888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332997888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.434961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.434963:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332997888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827798.434967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827798.434980:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.434986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de0e00 x1874663332998016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.434994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332998016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.434999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.435002:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.435004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827798.435008:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827798.435018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.435025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de0a80 x1874663332998144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.435032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332998144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.435037:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.435039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.435040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827798.435044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827798.435054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827798.435060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827798/real 1787827798] req@ffff9057d0de0000 x1874663332998272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827798.435066:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332998272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827798.435072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827798.435073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827798.435076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332998272:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827798.818391:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827799.842438:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827800.868457:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827802.914487:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827803.554526:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332998400:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827803.554629:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.554697:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de1880 x1874663332998400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.554725:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663332998400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.554765:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.554779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.554790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332998400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827803.554822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827803.554847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.554856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de1180 x1874663332998528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.554865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663332998528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.554871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.554873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.554875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827803.554879:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827803.554893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.554899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de2a00 x1874663332998656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.554907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663332998656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.554912:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.554914:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.554916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827803.554920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827803.554932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.554938:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de0700 x1874663332998784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.554945:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663332998784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.554950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.554952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.554954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827803.554957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827803.554968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.554973:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de0000 x1874663332998912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.554985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663332998912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.554990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.554992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.554994:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332998912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827803.554997:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827803.555008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.555013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de0a80 x1874663332999040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.555020:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663332999040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.555025:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.555027:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.555029:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827803.555033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827803.555054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.555061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de0e00 x1874663332999168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.555067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663332999168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.555077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.555079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.555082:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827803.555085:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827803.555096:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827803.555102:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827803/real 1787827803] req@ffff9057d0de3100 x1874663332999296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827803.555109:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663332999296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827803.555115:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827803.555117:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827803.555118:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663332999296:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827803.938491:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827804.962498:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827805.986436:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827807.010626:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827808.674373:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.674386:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827808.674392:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.674405:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.674431:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.674441:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827808.674490:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663332999424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827808.674591:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.674600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827808.674605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.674615:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.674625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.674635:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.674648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827808.674674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663332999552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827808.674690:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.674695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827808.674698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.674703:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.674708:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.674715:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.674722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827808.674734:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332999424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787827808.674754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663332999680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827808.674791:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.674800:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827808.674805:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.674817:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.674826:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.674836:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.674850:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827808.674878:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663332999808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827808.674890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787827808.674911:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.674920:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827808.674925:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.674936:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.674946:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.674955:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.674980:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827808.674998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb222300 x1874663332999424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827808.675011:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663332999936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827808.675060:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.675065:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827808.675070:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827808.675072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663332999424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827808.675083:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.675130:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827808.675146:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.675162:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827808.675172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.675186:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787827808.675191:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333000064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827808.675204:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663332999424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787827808.675313:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.675320:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827808.675324:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.675334:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.675341:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827808.675350:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787827808.675351:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.675371:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827808.675403:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333000192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827808.675421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787827808.675430:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827808.675436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827808.675440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827808.675450:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9619 00000100:00080000:3.0:1787827808.675457:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827808.675464:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb220380 x1874663332999552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827808.675471:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827808.675493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827808.675531:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663332999552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827808.675534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333000320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827808.675567:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.675579:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.675594:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827808.675660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827808.675707:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.675732:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb220700 x1874663332999680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.675765:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663332999680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.675775:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.675778:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.675782:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827808.675788:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827808.677504:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.677526:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb221880 x1874663332999808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.677543:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663332999808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.677554:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.677559:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.677563:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827808.677572:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827808.677596:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.677609:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb223100 x1874663332999936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.677632:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663332999936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.677648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.677655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.677662:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663332999936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827808.677674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827808.677714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.677749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb222d80 x1874663333000064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.677762:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333000064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.677772:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.677775:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.677779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827808.677785:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827808.677806:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.677816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb222a00 x1874663333000192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.677827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333000192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.677835:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.677838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.677841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827808.677846:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827808.677862:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827808.677871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff9057fb221c00 x1874663333000320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827808.677882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333000320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827808.677890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827808.677893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827808.677897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333000320:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827809.058479:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827810.082510:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827811.106475:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827813.154430:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827813.794396:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794410:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827813.794416:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794430:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794444:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794454:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827813.794503:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333000448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794551:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827813.794554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794560:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794565:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794569:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827813.794596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333000576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794613:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827813.794615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827813.794651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333000704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794665:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827813.794680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794693:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794697:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794715:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827813.794740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333000832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794753:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827813.794760:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794764:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794768:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794772:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794778:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827813.794789:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333000960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794800:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794804:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827813.794806:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794811:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794815:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794818:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794824:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827813.794841:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333001088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794878:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794882:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827813.794884:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.794890:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.794895:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.794906:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.794920:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827813.794947:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333001216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827813.794991:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827813.794997:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827813.795005:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827813.795019:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9624 00000100:00080000:1.0:1787827813.795027:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827813.795041:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827813.795073:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827813.795092:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333001344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827814.178649:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827815.202453:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827816.226423:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827817.250591:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827818.914366:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914380:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827818.914386:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914399:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914411:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914419:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827818.914467:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333001472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827818.914517:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914538:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914544:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914550:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827818.914582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333001600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827818.914602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914606:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827818.914632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333001728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827818.914648:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914651:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914654:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827818.914670:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333001856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827818.914696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914699:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914704:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827818.914716:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333001984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914735:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914739:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827818.914740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914746:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914751:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914764:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914771:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827818.914790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333002112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914814:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914825:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827818.914828:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914841:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914853:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914861:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914872:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827818.914894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333002240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827818.914910:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827818.914923:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827818.914926:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827818.914933:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9629 00000100:00080000:3.0:1787827818.914938:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827818.914942:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827818.914948:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827818.914969:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333002368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827819.298426:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827820.322558:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827821.346575:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827823.394495:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827824.034716:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333002496:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827824.034814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.034953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de1500 x1874663333002496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333002496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035123:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333002496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827824.035156:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827824.035301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035333:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de1f80 x1874663333002624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035392:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333002624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035424:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035434:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827824.035472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827824.035513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035528:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de1c00 x1874663333002752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035554:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333002752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035564:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035569:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035573:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827824.035581:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827824.035602:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de3800 x1874663333002880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035638:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333002880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035654:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035660:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035667:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333002880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827824.035680:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827824.035719:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035738:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de3100 x1874663333003008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035753:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333003008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035765:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035769:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827824.035781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827824.035805:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de0e00 x1874663333003136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035836:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333003136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035860:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.035873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827824.035886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827824.035925:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.035945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de0a80 x1874663333003264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.035960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333003264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.035971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.035976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.036006:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827824.036016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827824.036045:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827824.036059:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9057d0de0000 x1874663333003392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827824.036082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333003392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827824.036099:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827824.036106:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827824.036113:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333003392:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827824.418394:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827825.442383:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827826.466388:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827827.490347:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827829.154607:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333003520:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827829.154710:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.154791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de0700 x1874663333003520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.154860:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333003520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.154917:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.154926:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.154938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333003520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827829.154989:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827829.155033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de2a00 x1874663333003648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155095:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333003648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155115:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155124:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827829.155133:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827829.155152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155162:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de1180 x1874663333003776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155173:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333003776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827829.155284:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827829.155312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155326:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de1880 x1874663333003904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155340:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333003904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155353:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155357:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333003904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827829.155363:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333004032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827829.155381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155391:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de0000 x1874663333004032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155404:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333004032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155416:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155419:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333004032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827829.155425:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827829.155442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155451:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de0a80 x1874663333004160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155463:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333004160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155476:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155480:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827829.155485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827829.155507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155516:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de0e00 x1874663333004288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155528:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333004288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155539:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155542:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827829.155548:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827829.155565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827829.155574:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9057d0de3100 x1874663333004416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827829.155586:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333004416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.155594:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827829.155597:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827829.155602:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333004416:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827829.538346:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827830.562424:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827831.586420:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827833.634388:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827834.274397:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827834.274412:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274421:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274432:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274440:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827834.274491:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333004544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827834.274528:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274532:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274545:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274549:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827834.274577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333004672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827834.274603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274608:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274611:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274615:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827834.274632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333004800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274640:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827834.274645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274648:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274651:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274655:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274659:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827834.274666:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333004928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274674:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827834.274678:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274681:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274683:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274686:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274689:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827834.274696:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333005056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274703:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827834.274707:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274711:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274713:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274716:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827834.274737:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333005184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274744:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827834.274748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274750:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274753:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274755:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274758:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827834.274764:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333005312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827834.274772:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827834.274775:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827834.274777:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827834.274779:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9645 00000100:00080000:1.0:1787827834.274781:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827834.274784:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827834.274798:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827834.274816:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333005440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827834.658359:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827835.682412:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827836.706365:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827837.730444:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827839.394359:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827839.394395:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394407:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394422:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394431:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827839.394473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333005568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394508:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827839.394514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394518:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394522:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394527:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827839.394548:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333005696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827839.394566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394569:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394572:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394575:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827839.394595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333005824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827839.394625:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394629:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394646:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394651:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827839.394695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333005952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394707:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394710:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827839.394712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394716:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394721:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394726:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394748:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827839.394767:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333006080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394797:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394801:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827839.394804:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394817:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394830:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394847:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394862:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827839.394918:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333006208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.394949:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.394954:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827839.394958:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.394971:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.394978:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.394984:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.394991:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827839.395010:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333006336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827839.395027:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827839.395033:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827839.395036:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827839.395046:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9650 00000100:00080000:3.0:1787827839.395052:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827839.395061:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827839.395070:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827839.395100:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333006464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827839.778467:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827840.802362:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827841.826345:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827843.874415:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827844.514463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333006592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827844.514582:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827844.514623:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827844.514659:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827844.514694:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827844.514730:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827844.514753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827844.514771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007488:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827844.898396:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827845.922442:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827846.946389:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827847.586659:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827847.586738:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827847.586759:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827847.586862:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827847.586887:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827847.586908:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827847.586911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de1c00 x1874663333006720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827847.586927:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827847.586952:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827847.586982:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333006720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827847.586983:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827847.587009:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00080000:1.0:1787827847.587015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827847.587058:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de1f80 x1874663333006848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333006848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587109:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587133:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827847.587143:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de1500 x1874663333006976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587194:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333006976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587212:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587217:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333006976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827847.587262:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de3100 x1874663333007104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587280:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333007104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587321:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587329:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827847.587341:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de3800 x1874663333006592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333006592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587377:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587381:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333006592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827847.587416:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de0e00 x1874663333007232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587434:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333007232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587449:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587454:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827847.587462:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de0a80 x1874663333007360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587477:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333007360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587490:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587495:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827847.587504:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827844/real 1787827847] req@ffff9057d0de0000 x1874663333007488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827847.587519:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333007488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827847.587530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827847.587534:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827847.587538:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333007488:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827847.970395:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827849.634586:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333007616:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827849.634687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.634744:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de1880 x1874663333007616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.634778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333007616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.634798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.634835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.634844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333007616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827849.634907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827849.634951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.634977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de1180 x1874663333007744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635016:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333007744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635042:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827849.635062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827849.635098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635119:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de2a00 x1874663333007872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635143:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333007872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635165:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333007872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827849.635184:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827849.635267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635295:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de0700 x1874663333008000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635325:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333008000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635360:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635367:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827849.635379:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827849.635418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635440:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de0000 x1874663333008128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635466:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333008128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635489:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635495:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827849.635529:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827849.635569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635591:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de0a80 x1874663333008256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635612:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333008256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635626:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635631:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635638:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827849.635649:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827849.635683:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635704:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de0e00 x1874663333008384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635727:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333008384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635772:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827849.635783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827849.635855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827849.635881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff9057d0de3800 x1874663333008512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827849.635906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333008512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827849.635923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827849.635929:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827849.635935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333008512:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827850.018656:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827851.042423:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827852.066434:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827854.114335:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827854.754565:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333008640:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827854.754692:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.754820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de3100 x1874663333008640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.754876:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333008640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.754922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.754936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.754953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333008640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827854.754990:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827854.755040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de1500 x1874663333008768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333008768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755095:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755099:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827854.755106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827854.755125:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755135:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de1f80 x1874663333008896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755155:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333008896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755167:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755170:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333008896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827854.755176:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333009024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827854.755192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755202:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de1c00 x1874663333009024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333009024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755285:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333009024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827854.755292:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333009152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827854.755315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755327:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de3800 x1874663333009152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755340:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333009152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755359:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755362:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333009152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827854.755368:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827854.755386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755396:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de0e00 x1874663333009280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755408:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333009280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755417:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755419:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755422:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827854.755442:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827854.755460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755470:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de0a80 x1874663333009408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755482:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333009408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755495:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755502:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827854.755512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827854.755542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827854.755561:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff9057d0de0000 x1874663333009536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827854.755582:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333009536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827854.755593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827854.755596:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827854.755600:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333009536:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827855.138365:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827856.162366:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827857.186536:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827858.210493:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827859.874367:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874380:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827859.874385:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874398:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874412:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874421:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827859.874472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333009664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827859.874515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874519:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874523:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874529:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827859.874553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333009792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827859.874589:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874594:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874598:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874603:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827859.874632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333009920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827859.874647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874652:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874658:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874663:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827859.874687:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333010048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874702:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827859.874709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874727:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874754:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874760:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874768:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827859.874800:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333010176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874814:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874819:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827859.874822:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874829:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874836:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874849:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874868:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827859.874895:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333010304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.874933:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.874939:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827859.874943:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.874962:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.874970:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.874984:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.874996:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827859.875020:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333010432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827859.875061:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827859.875075:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827859.875079:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827859.875090:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9670 00000100:00080000:1.0:1787827859.875101:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827859.875118:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827859.875134:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827859.875163:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333010560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827860.258408:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827861.282355:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827862.306407:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827864.354425:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827864.994329:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994341:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827864.994346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994355:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994365:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994371:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827864.994405:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333010688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994428:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994431:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827864.994432:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994443:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994454:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.994463:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333010816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.994489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994492:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994494:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994497:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.994508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333010944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827864.994519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994521:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994530:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994532:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827864.994544:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333011072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.994557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994560:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827864.994592:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333011200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827864.994617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994625:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994634:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994642:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.994668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333011328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994679:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.994685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994688:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994708:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994716:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.994735:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333011456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.994762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.994769:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.994771:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.994779:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9675 00000100:00080000:3.0:1787827864.994784:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.994791:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827864.994795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827864.994807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333011584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827865.378349:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827866.402374:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827867.426335:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827868.450367:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827870.114375:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.114390:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827870.114397:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.114413:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.114432:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.114443:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827870.114514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333011712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.114601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.114606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827870.114609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.114616:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.114622:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.114628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.114644:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827870.114713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333011840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.114731:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.114738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827870.114768:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.114777:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.114783:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.114791:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.114801:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827870.114853:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333011968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.114867:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.114878:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827870.114884:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.114894:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.114923:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.114931:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.114939:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827870.114971:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333012096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.115008:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.115038:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827870.115042:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.115059:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.115075:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.115096:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.115125:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827870.115168:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333012224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.115193:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.115199:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827870.115202:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.115234:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.115247:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.115328:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.115340:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827870.115365:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333012352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.115407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.115418:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827870.115442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.115454:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.115459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.115470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.115487:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827870.115532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333012480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827870.115577:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827870.115604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827870.115616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827870.115625:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9680 00000100:00080000:1.0:1787827870.115640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827870.115711:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827870.115737:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827870.115782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333012608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827870.498443:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827871.522395:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827872.546388:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827874.594374:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827875.234683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333012736:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827875.234782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.234893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de0700 x1874663333012736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.234922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333012736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.234936:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.234940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.234947:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333012736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827875.234969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333012864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827875.234995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de2a00 x1874663333012864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333012864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333012864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827875.235055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333012992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827875.235079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de1180 x1874663333012992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333012992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235115:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333012992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827875.235120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333013120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827875.235131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de1880 x1874663333013120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235144:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333013120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333013120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827875.235156:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333013248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827875.235165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235171:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de0000 x1874663333013248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235178:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333013248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235184:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235187:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333013248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827875.235190:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827875.235201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de0a80 x1874663333013376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235213:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333013376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235237:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235239:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827875.235251:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827875.235264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235271:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de0e00 x1874663333013504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235278:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333013504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235285:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235287:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827875.235291:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827875.235300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827875.235306:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827875/real 1787827875] req@ffff9057d0de3800 x1874663333013632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827875.235313:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333013632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827875.235318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827875.235320:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827875.235322:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333013632:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827875.618403:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827876.642374:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827877.666457:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827878.690595:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827880.354299:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354309:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827880.354314:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354324:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354335:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827880.354378:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333013760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354401:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354404:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827880.354405:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354408:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354411:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354415:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354423:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827880.354436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333013888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354445:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827880.354449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354451:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354454:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354457:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354460:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827880.354468:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333014016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827880.354477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354480:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354482:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354487:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827880.354501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333014144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354509:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827880.354510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354513:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354517:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354519:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827880.354534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333014272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827880.354550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354559:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354562:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354567:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827880.354583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333014400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354606:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827880.354610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354617:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354621:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354625:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827880.354643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333014528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827880.354653:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827880.354662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827880.354665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827880.354671:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9691 00000100:00080000:1.0:1787827880.354677:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827880.354682:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827880.354688:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827880.354702:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333014656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827880.738377:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827881.762355:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827882.786358:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827884.834362:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827885.474312:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474321:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827885.474325:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474333:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474342:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474347:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827885.474378:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333014784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474399:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474403:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827885.474405:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474410:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474414:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474419:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474428:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827885.474441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333014912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827885.474455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474458:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474461:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827885.474478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333015040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474489:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827885.474490:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474493:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474496:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474499:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827885.474512:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333015168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827885.474530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474537:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474541:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474544:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827885.474568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333015296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474577:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474581:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827885.474583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474587:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474590:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474594:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474598:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827885.474608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333015424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827885.474622:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474626:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474633:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827885.474648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333015552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827885.474658:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827885.474662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827885.474664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827885.474667:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9696 00000100:00080000:3.0:1787827885.474670:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827885.474673:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827885.474678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827885.474687:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333015680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827885.858376:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827886.882373:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827887.906336:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827888.930341:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827890.594334:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827890.594346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594354:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594363:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594369:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827890.594400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333015808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594421:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594424:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827890.594425:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594428:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594431:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594433:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594441:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827890.594452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333015936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827890.594468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594477:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594483:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827890.594492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333016064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594499:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827890.594503:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594506:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594511:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594514:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827890.594535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333016192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827890.594558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594568:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827890.594580:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333016320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594586:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827890.594594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594602:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594608:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594612:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827890.594621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333016448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594636:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594641:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827890.594644:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594650:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594653:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594658:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827890.594668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333016576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827890.594678:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827890.594680:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827890.594681:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827890.594684:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9701 00000100:00080000:1.0:1787827890.594687:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827890.594694:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827890.594699:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827890.594711:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333016704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827890.978352:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827892.002367:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827893.026444:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827895.074363:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827895.714443:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333016832:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827895.714521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714568:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de1c00 x1874663333016832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.714606:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333016832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.714641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.714653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.714664:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333016832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827895.714694:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333016960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827895.714735:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de1f80 x1874663333016960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.714763:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333016960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.714772:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.714777:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.714781:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333016960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827895.714788:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827895.714810:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714822:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de1500 x1874663333017088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.714835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333017088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.714844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.714847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.714851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827895.714857:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827895.714876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de3100 x1874663333017216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.714896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333017216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.714902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.714904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.714906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827895.714910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827895.714928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de3800 x1874663333017344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.714947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333017344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.714953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.714955:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.714958:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827895.714964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827895.714984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.714996:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de0e00 x1874663333017472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.715018:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333017472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.715028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.715032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.715035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827895.715041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827895.715062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.715073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de0a80 x1874663333017600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.715085:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333017600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.715094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.715097:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.715101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827895.715120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827895.715136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827895.715146:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff9057d0de0000 x1874663333017728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827895.715158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333017728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827895.715166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827895.715168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827895.715171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333017728:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827896.098369:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827897.122333:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827898.146316:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827899.170335:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827900.834478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333017856:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827900.834552:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.834600:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de1880 x1874663333017856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.834633:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333017856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.834662:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.834669:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.834675:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333017856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827900.834706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827900.834737:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.834754:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de1180 x1874663333017984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.834776:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333017984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.834791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.834799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.834809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333017984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827900.834824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827900.834861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.834875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de2a00 x1874663333018112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.834900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333018112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.834908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.834911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.834913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827900.834919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827900.834933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.834939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de0700 x1874663333018240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.834947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333018240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.834952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.834954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.834956:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827900.834959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827900.834969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.834975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de0000 x1874663333018368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.834987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333018368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.834992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.834993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.834995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333018368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827900.835007:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827900.835017:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.835023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de0a80 x1874663333018496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.835030:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333018496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.835034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.835036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.835038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827900.835042:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827900.835051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.835057:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de0e00 x1874663333018624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.835066:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333018624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.835081:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.835084:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.835088:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827900.835095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827900.835115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827900.835126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff9057d0de3800 x1874663333018752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827900.835135:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333018752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827900.835141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827900.835143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827900.835145:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333018752:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827901.218429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827902.242393:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827903.266371:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827905.314343:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827905.954349:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954359:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827905.954363:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.954373:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827905.954444:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333018880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827905.954490:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.954493:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954496:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954499:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954509:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.954519:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333019008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827905.954532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827905.954536:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333018880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787827905.954537:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954544:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954548:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.954561:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333019136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827905.954579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.954583:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954587:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954591:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827905.954603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333019264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827905.954613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.954615:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954618:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827905.954623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787827905.954624:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827905.954637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333019392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954645:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827905.954650:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.954653:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954656:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954658:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.954668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333019520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.954679:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954682:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827905.954684:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827905.954687:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb223800 x1874663333018880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827905.954690:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.954706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.954723:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333019648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827905.954727:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333018880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827905.954745:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.954749:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827905.954750:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827905.954751:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827905.954758:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787827905.954759:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9716 00000100:00080000:3.0:1787827905.954764:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827905.954765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333018880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787827905.954771:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827905.954774:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827905.954782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333019776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827905.954797:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827905.954826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.954836:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb222680 x1874663333019008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.954845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333019008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.954852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.954854:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.954856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827905.954861:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827905.954873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.954879:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb220e00 x1874663333019136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.954898:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333019136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.954903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.954905:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.954907:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827905.954911:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827905.954923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.954935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb220a80 x1874663333019264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.954944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333019264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.954949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.954951:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.954953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827905.954957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827905.954968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.954974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb221180 x1874663333019392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.954982:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333019392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.954987:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.954989:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.954991:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333019392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827905.954994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827905.955005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.955010:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb220000 x1874663333019520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.955023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333019520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.955029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.955031:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.955034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827905.955037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827905.955049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.955055:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb223b80 x1874663333019648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.955068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333019648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.955074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.955076:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.955078:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827905.955081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827905.955092:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827905.955098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff9057fb221500 x1874663333019776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827905.955105:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333019776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827905.955110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827905.955112:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827905.955115:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333019776:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827906.338396:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827907.362418:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827908.386420:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827909.410360:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827911.074485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333019904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827911.074595:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827911.074636:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827911.074662:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827911.074726:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827911.074804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827911.074845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827911.074953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020800:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827911.458383:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827912.482382:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827913.506338:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827914.146485:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787827914.146524:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827914.146538:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146583:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146594:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146607:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146619:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146630:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827914.146632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de1c00 x1874663333020288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827914.146646:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787827914.146666:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787827914.146667:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333020288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146689:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827914.146702:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de1500 x1874663333020032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146716:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333020032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146723:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146726:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827914.146736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de1f80 x1874663333020160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333020160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146759:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827914.146765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de3100 x1874663333019904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146777:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333019904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333019904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827914.146818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de0000 x1874663333020800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333020800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146840:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146843:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146847:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827914.146853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de0a80 x1874663333020672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146864:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333020672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146874:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146878:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827914.146884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de0e00 x1874663333020544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333020544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146905:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333020544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827914.146914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827914] req@ffff9057d0de3800 x1874663333020416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827914.146925:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333020416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827914.146932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827914.146935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827914.146938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333020416:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827915.554412:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827916.194491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333020928:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827916.194562:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.194604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de0700 x1874663333020928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.194636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333020928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.194660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.194671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.194679:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333020928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827916.194708:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827916.194766:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.194788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de2a00 x1874663333021056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.194812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333021056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.194855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.194866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.194874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827916.194889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827916.194923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.194940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de1180 x1874663333021184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.194961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333021184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.194969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.194973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.194976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827916.194982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827916.195003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.195015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de1880 x1874663333021312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.195026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333021312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.195034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.195036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.195040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827916.195056:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827916.195076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.195089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de3800 x1874663333021440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.195099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333021440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.195106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.195109:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.195112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333021440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827916.195118:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827916.195137:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.195148:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de0e00 x1874663333021568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.195159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333021568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.195167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.195170:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.195173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827916.195179:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827916.195199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.195211:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de0a80 x1874663333021696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.195270:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333021696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.195279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.195282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.195286:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827916.195294:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827916.195318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827916.195330:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9057d0de0000 x1874663333021824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.195360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333021824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.195370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827916.195373:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827916.195377:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333021824:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827916.578331:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827917.602353:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827918.626338:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827919.650348:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827921.314501:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333021952:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827921.314562:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314600:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de3100 x1874663333021952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314621:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333021952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314639:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314656:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314662:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333021952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827921.314672:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827921.314697:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314710:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de1f80 x1874663333022080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314721:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333022080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314730:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314738:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314745:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827921.314783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827921.314802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de1500 x1874663333022208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333022208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314825:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827921.314834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827921.314844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de1c00 x1874663333022336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333022336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314864:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827921.314869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827921.314878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de0000 x1874663333022464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314890:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333022464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333022464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827921.314901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827921.314910:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de0a80 x1874663333022592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333022592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827921.314934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827921.314944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de0e00 x1874663333022720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333022720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.314960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.314962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.314964:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827921.314973:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827921.314983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827921.314989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827921/real 1787827921] req@ffff9057d0de3800 x1874663333022848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827921.314996:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333022848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827921.315006:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827921.315008:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827921.315019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333022848:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827921.698537:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827922.722399:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827923.746503:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827925.794464:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827926.434587:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333022976:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827926.434720:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.434798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de1880 x1874663333022976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.434831:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333022976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.434865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.434882:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.434890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333022976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827926.434933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827926.434999:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de1180 x1874663333023104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333023104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435120:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827926.435178:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827926.435234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435338:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de2a00 x1874663333023232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435351:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333023232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435363:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435376:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827926.435383:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827926.435405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435416:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de0700 x1874663333023360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435428:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333023360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435439:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435442:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827926.435448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827926.435464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435473:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de3800 x1874663333023488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435484:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333023488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435495:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435498:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333023488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827926.435527:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827926.435550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435561:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de0e00 x1874663333023616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435573:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333023616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435581:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435584:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435587:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827926.435592:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827926.435634:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435653:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de0a80 x1874663333023744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435692:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333023744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435723:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435729:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827926.435739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827926.435765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827926.435777:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827926/real 1787827926] req@ffff9057d0de0000 x1874663333023872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827926.435790:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333023872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827926.435799:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827926.435802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827926.435806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333023872:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827926.818422:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827927.842420:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827928.866758:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827929.890449:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827931.554470:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333024000:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827931.554534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554575:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de1c00 x1874663333024000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554601:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333024000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554621:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554628:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554637:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333024000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827931.554649:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827931.554672:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554684:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de1500 x1874663333024128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554696:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333024128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554712:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827931.554741:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827931.554756:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554763:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de1f80 x1874663333024256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554770:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333024256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554775:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554777:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827931.554782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827931.554793:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de3100 x1874663333024384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554806:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333024384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827931.554818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827931.554827:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554833:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de0000 x1874663333024512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333024512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333024512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827931.554851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827931.554861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554867:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de0a80 x1874663333024640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554874:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333024640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554879:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554881:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554883:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827931.554886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827931.554896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554901:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de0e00 x1874663333024768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554916:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333024768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827931.554929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827931.554940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827931.554946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827931/real 1787827931] req@ffff9057d0de3800 x1874663333024896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827931.554952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333024896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827931.554958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827931.554959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827931.554962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333024896:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827931.938418:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827932.962365:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827933.986488:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827936.034726:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827936.674354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827936.674369:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674382:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674395:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674403:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827936.674456:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333025024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674504:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827936.674509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674514:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674517:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674520:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827936.674548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333025152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674558:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827936.674560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674563:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674566:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674572:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827936.674594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333025280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674602:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674605:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827936.674606:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674614:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674620:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674624:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674628:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827936.674642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333025408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827936.674656:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674658:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674660:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674663:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827936.674678:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333025536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674684:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674688:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827936.674690:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674694:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674698:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674701:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674704:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827936.674711:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333025664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674717:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674719:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827936.674720:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674725:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674729:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674731:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674734:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827936.674753:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333025792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827936.674775:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827936.674780:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827936.674782:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827936.674786:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9747 00000100:00080000:1.0:1787827936.674789:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827936.674794:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827936.674800:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827936.674808:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333025920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827937.058393:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827938.082502:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827939.106528:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827940.130366:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827941.794288:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794309:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827941.794312:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794320:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794329:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794335:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827941.794371:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333026048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794427:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794432:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827941.794435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794450:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827941.794487:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333026176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794504:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827941.794506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794513:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794518:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827941.794536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333026304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827941.794552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794557:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794560:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794569:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827941.794579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333026432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794593:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827941.794595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794599:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794603:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794608:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827941.794624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333026560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794636:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827941.794643:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794647:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794651:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827941.794671:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333026688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827941.794689:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794694:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794698:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794702:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827941.794720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333026816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827941.794732:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827941.794737:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827941.794739:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827941.794743:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9752 00000100:00080000:3.0:1787827941.794747:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827941.794752:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827941.794757:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827941.794768:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333026944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827942.178390:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827943.202500:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827944.226389:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827946.274378:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827946.914509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333027072:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827946.914579:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.914639:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de0700 x1874663333027072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.914661:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333027072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.914681:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.914688:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.914696:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333027072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827946.914708:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827946.914732:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.914747:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de2a00 x1874663333027200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.914778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333027200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.914790:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.914795:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.914799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827946.914824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827946.914842:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.914877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de1180 x1874663333027328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.914886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333027328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.914892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.914894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.914897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827946.914901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827946.914916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.914923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de1880 x1874663333027456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.914931:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333027456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.914936:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.914938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.914940:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827946.914943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827946.914954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.914961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de3800 x1874663333027584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.914968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333027584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.914974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.914976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.914978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333027584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827946.914982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827946.914993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.915000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de0e00 x1874663333027712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.915008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333027712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.915013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.915014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.915016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827946.915020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827946.915031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.915037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de0a80 x1874663333027840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.915044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333027840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.915050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.915052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.915054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827946.915058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827946.915068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827946.915074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff9057d0de0000 x1874663333027968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827946.915081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333027968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827946.915089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827946.915092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827946.915096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333027968:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827947.298376:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827948.322354:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827949.346423:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827950.370481:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827952.034485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333028096:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827952.034574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.034632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de3100 x1874663333028096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.034692:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333028096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.034712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.034725:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.034737:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333028096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827952.034786:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827952.034830:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.034860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de1f80 x1874663333028224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.034869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333028224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.034875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.034877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.034880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.034884:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827952.034895:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.034901:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de1500 x1874663333028352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.034908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333028352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.034932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.034935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.034938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.034943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827952.034958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.034967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de1c00 x1874663333028480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.034977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333028480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.034984:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.034986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.034989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827952.034994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827952.035007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.035015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de0000 x1874663333028608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.035024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333028608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.035031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.035034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.035036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333028608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827952.035041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827952.035054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.035061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de0a80 x1874663333028736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.035070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333028736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.035078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.035081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.035084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.035088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827952.035130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.035161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de0e00 x1874663333028864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.035168:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333028864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.035173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.035175:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.035177:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.035190:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827952.035201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827952.035207:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9057d0de3800 x1874663333028992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.035214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333028992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.035219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827952.035221:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827952.035223:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333028992:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827952.418481:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827953.442402:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827954.466395:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827956.514395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827957.154586:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333029120:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787827957.154701:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.154776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de1880 x1874663333029120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.154858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333029120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.154914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.154929:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.154947:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333029120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787827957.154995:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827957.155052:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de1180 x1874663333029248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155177:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333029248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155197:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155202:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827957.155210:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827957.155280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155303:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de2a00 x1874663333029376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155318:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333029376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155336:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827957.155343:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827957.155374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155394:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de0700 x1874663333029504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155420:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333029504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155443:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827957.155460:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827957.155494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155508:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de3800 x1874663333029632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155530:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333029632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155540:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155544:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155547:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333029632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827957.155554:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333029760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827957.155584:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de0e00 x1874663333029760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155627:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333029760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155649:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155656:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333029760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827957.155684:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333029888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827957.155712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155732:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de0a80 x1874663333029888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155746:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333029888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155762:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155770:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333029888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827957.155777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827957.155819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827957.155839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9057d0de0000 x1874663333030016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827957.155861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333030016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827957.155877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827957.155882:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827957.155888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030016:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787827957.538494:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827958.562392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827959.586422:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827960.610440:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827962.274421:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827962.274441:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274455:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274483:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274498:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827962.274564:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333030144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827962.274610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827962.274617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274622:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274628:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274634:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.274647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827962.274666:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333030272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827962.274677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333030144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787827962.274685:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274691:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827962.274693:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274697:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274701:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.274712:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827962.274728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333030400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827962.274738:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274743:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827962.274745:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274750:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274756:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274759:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.274765:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827962.274776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333030528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827962.274786:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274792:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827962.274797:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827962.274798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787827962.274804:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274809:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274814:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.274821:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827962.274834:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333030656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827962.274853:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274857:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827962.274859:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274863:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274868:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.274872:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.274878:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827962.274881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb221c00 x1874663333030144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827962.274894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333030784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827962.274921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333030144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.274937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.274943:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.274951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333030144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787827962.274951:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.274965:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827962.274968:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.274977:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.274985:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827962.274992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787827962.275000:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827962.275013:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827962.275056:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333030912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827962.275060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787827962.275083:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827962.275089:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827962.275096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb222a00 x1874663333030272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827962.275097:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827962.275112:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9773 00000100:00080000:3.0:1787827962.275126:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827962.275136:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275143:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333030272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827962.275149:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827962.275161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275174:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787827962.275182:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333031040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827962.275194:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827962.275314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275330:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb222d80 x1874663333030400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275344:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333030400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275358:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275362:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827962.275368:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827962.275390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275402:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb223100 x1874663333030528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275414:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333030528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275438:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275441:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827962.275448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827962.275468:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275478:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb221880 x1874663333030656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275490:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333030656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275502:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275505:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333030656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787827962.275510:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827962.275541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275552:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb220700 x1874663333030784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275564:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333030784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275577:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275580:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827962.275603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787827962.275621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb220380 x1874663333030912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275643:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333030912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275652:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275658:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333030912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787827962.275664:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333031040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787827962.275680:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787827962.275690:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827962/real 1787827962] req@ffff9057fb222300 x1874663333031040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827962.275707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333031040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827962.275721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827962.275726:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787827962.275734:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333031040:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787827962.658566:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827963.682364:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827964.706717:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827966.754496:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827967.394385:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394401:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827967.394408:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394425:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394442:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394455:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827967.394514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333031168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394576:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827967.394580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394589:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394601:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.394613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827967.394637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333031296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394656:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394673:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827967.394676:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394681:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394685:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394692:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.394698:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827967.394715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333031424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394727:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394731:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827967.394735:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394740:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394744:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394748:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.394754:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827967.394774:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333031552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394793:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394798:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827967.394801:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394806:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394811:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394815:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.394820:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827967.394834:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333031680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394846:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394854:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827967.394861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.394872:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.394881:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.394892:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.394898:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827967.394916:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333031808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.394958:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.394971:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827967.394974:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.395010:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.395020:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.395042:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.395050:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827967.395072:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333031936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827967.395098:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827967.395102:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827967.395105:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827967.395110:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9778 00000100:00080000:1.0:1787827967.395116:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827967.395120:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827967.395128:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827967.395148:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333032064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827967.778435:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827968.802483:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827969.826523:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827970.850514:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827972.514399:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514411:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827972.514417:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514430:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514444:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514453:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827972.514502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333032192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514537:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514542:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827972.514545:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514550:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514555:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514559:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827972.514587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333032320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514603:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514609:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827972.514612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514624:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514629:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827972.514651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333032448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514671:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827972.514673:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514678:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514683:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514688:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514695:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827972.514708:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333032576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514725:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827972.514732:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514742:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514749:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514754:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827972.514777:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333032704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514793:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827972.514796:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514802:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514809:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514820:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514833:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827972.514864:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333032832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.514909:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.514914:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827972.514916:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.514926:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.514932:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.514945:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.514984:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827972.515013:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333032960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827972.515030:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827972.515038:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827972.515043:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827972.515056:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9783 00000100:00080000:3.0:1787827972.515062:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827972.515066:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827972.515074:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827972.515091:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333033088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827972.898492:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827973.922392:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827974.946464:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827976.994381:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827977.634386:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634403:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827977.634421:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634438:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634457:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634469:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787827977.634525:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333033216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634567:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827977.634574:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634580:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634586:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634591:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634603:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827977.634620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333033344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634635:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634640:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827977.634642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634647:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634650:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634658:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634667:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827977.634687:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333033472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634701:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634707:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827977.634710:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634715:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634719:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634723:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634729:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827977.634759:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333033600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634772:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634775:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827977.634777:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634782:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634787:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634791:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634797:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827977.634817:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333033728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634840:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634846:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827977.634849:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634856:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634862:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634868:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634877:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827977.634899:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333033856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.634946:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.634951:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827977.634955:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.634962:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.634974:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.634989:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.634999:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827977.635022:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333033984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827977.635054:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827977.635060:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827977.635064:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827977.635071:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9788 00000100:00080000:0.0:1787827977.635077:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827977.635083:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787827977.635091:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827977.635111:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333034112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827978.018458:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827979.042435:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827980.066494:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827980.706574:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787827980.706644:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827980.706682:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706739:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706755:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706769:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706813:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706828:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706843:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787827980.706876:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787827981.090512:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827982.754481:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.754493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827982.754498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.754511:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.754524:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.754533:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827982.754578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333034240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.754619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.754626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827982.754629:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.754638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.754645:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.754654:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.754670:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827982.754699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333034368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.754732:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.754740:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827982.754744:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.754750:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.754758:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.754765:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.754776:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827982.754800:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333034496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.754833:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.754838:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827982.754842:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.754851:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.754870:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.754909:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.754921:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827982.754957:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333034624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.755023:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.755035:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827982.755043:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.755056:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.755065:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.755073:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.755085:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827982.755121:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333034752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.755146:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.755158:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827982.755165:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.755185:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.755193:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.755206:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.755272:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827982.755299:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333034880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.755335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.755345:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827982.755350:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.755370:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.755382:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.755398:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.755414:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827982.755449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333035008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827982.755471:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827982.755477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827982.755482:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827982.755493:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9793 00000100:00080000:1.0:1787827982.755506:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827982.755517:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827982.755527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827982.755553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333035136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827983.138536:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827984.162428:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827985.186441:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827987.234458:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827987.874331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827987.874347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874357:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874370:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874380:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827987.874432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333035264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827987.874475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874480:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874485:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.874506:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.874534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333035392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.874570:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874611:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.874639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.874668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333035520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827987.874691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874700:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874709:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874735:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.874742:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827987.874754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333035648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874768:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874774:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.874779:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874808:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874820:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874826:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.874834:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827987.874892:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333035776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874916:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874921:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827987.874924:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.874931:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.874937:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.874943:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.874950:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.874969:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333035904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.874991:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.874999:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.875002:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.875014:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.875023:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.875031:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.875040:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.875056:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333036032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.875100:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.875108:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.875112:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.875124:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9798 00000100:00080000:3.0:1787827987.875130:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.875154:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827987.875162:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827987.875173:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333036160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827988.258479:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827989.282455:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827990.306467:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827991.330486:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827992.994320:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994331:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827992.994336:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994361:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994370:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787827992.994417:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333036288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827992.994457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994469:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.994493:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333036416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994505:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.994512:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994516:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994520:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994529:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.994557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333036544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994574:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827992.994576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994602:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994606:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994612:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827992.994632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333036672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994645:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.994651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994667:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994672:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994677:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994683:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827992.994700:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333036800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994715:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994735:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827992.994745:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994759:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994769:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994782:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.994848:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333036928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994864:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994868:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.994871:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994882:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994894:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.994901:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.994909:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.994928:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333037056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.994962:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.994968:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.994974:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.994983:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9803 00000100:00080000:3.0:1787827992.994992:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.995004:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787827992.995010:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827992.995023:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333037184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827993.378587:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827994.402445:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827995.426508:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827997.474509:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827998.114455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827998.114473:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114486:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114500:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.114510:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787827998.114561:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333037312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.114601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827998.114608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114614:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114618:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.114622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.114635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827998.114655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333037440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.114676:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827998.114685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114692:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114696:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.114701:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.114707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827998.114725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333037568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.114737:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114742:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827998.114745:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114749:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114753:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.114762:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.114805:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827998.114848:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333037696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.114862:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114869:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827998.114873:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114888:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114898:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.114907:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.114914:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827998.114931:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333037824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.114962:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.114973:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827998.114975:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.114981:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.114998:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.115009:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.115022:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827998.115044:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333037952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.115060:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.115067:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827998.115074:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.115088:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.115097:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.115104:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.115117:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827998.115141:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333038080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827998.115173:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827998.115178:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827998.115180:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827998.115186:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9808 00000100:00080000:1.0:1787827998.115190:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827998.115194:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787827998.115201:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827998.115267:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333038208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827998.498519:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827999.522474:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828000.546717:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828001.570550:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828003.234278:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234286:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828003.234289:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234298:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234307:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234313:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828003.234344:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333038336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234369:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234373:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828003.234375:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234380:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234384:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234390:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234410:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828003.234424:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333038464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828003.234438:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234443:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234449:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828003.234458:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333038592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828003.234468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234470:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234481:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234487:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234491:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828003.234501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333038720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828003.234516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234534:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234540:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234547:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828003.234559:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333038848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234579:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828003.234580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234584:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828003.234600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333038976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828003.234624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234629:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234633:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234641:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828003.234668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333039104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828003.234692:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828003.234697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828003.234701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828003.234707:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9813 00000100:00080000:1.0:1787828003.234710:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828003.234714:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828003.234719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828003.234755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333039232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828003.618383:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828004.642430:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828005.666481:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828007.714574:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828008.354379:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354392:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828008.354396:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354409:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354423:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354433:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828008.354481:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333039360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354518:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828008.354525:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354531:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354535:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354541:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354553:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828008.354568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333039488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354583:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828008.354593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354602:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828008.354637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333039616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828008.354657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354661:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354677:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354683:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828008.354740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333039744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354756:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828008.354759:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354764:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354770:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354776:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354783:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828008.354803:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333039872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354826:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354831:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828008.354833:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354843:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354855:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354866:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354879:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828008.354894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333040000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354920:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354933:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828008.354939:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.354946:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.354950:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.354954:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.354959:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828008.354970:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333040128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828008.354988:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828008.354994:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828008.354996:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828008.355003:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9819 00000100:00080000:3.0:1787828008.355007:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828008.355011:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828008.355018:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828008.355038:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333040256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828008.738480:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828009.762480:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828010.786477:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828011.810537:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828013.474697:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333040384:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828013.474809:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.474888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de1c00 x1874663333040384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.474932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333040384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.474973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.474993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333040384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828013.475080:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828013.475150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475211:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de1500 x1874663333040512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475313:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333040512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475334:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475338:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828013.475358:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828013.475405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475418:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de1f80 x1874663333040640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475430:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333040640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475480:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475488:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828013.475500:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828013.475543:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475573:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de3100 x1874663333040768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475597:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333040768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475608:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475612:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475616:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828013.475624:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828013.475650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475692:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de0000 x1874663333040896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333040896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475747:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475754:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333040896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828013.475761:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828013.475788:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475809:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de0a80 x1874663333041024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475822:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333041024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475839:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475845:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828013.475879:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828013.475899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de0e00 x1874663333041152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.475923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333041152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.475932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.475935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.475939:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828013.475945:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828013.475964:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828013.475982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828013/real 1787828013] req@ffff9057d0de3800 x1874663333041280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828013.476002:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333041280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828013.476018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828013.476023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828013.476031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333041280:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828013.858501:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828014.882544:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828015.906389:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828017.954529:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828018.594311:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594318:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828018.594322:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594331:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594339:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594346:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828018.594381:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333041408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594405:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828018.594409:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594412:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594415:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594417:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594425:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.594436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333041536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594446:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.594457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594465:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.594477:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333041664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828018.594488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594490:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594493:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594494:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828018.594504:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333041792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594511:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594515:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.594519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594525:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594527:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594530:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828018.594538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333041920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828018.594549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594554:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594559:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594565:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594573:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.594583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333042048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.594615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594622:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594646:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594651:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.594659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333042176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.594674:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.594677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.594679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.594683:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9829 00000100:00080000:1.0:1787828018.594686:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.594693:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828018.594699:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828018.594710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333042304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828018.978377:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828020.002392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828021.026436:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828022.050594:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828023.714317:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828023.714328:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714337:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714346:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714351:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828023.714384:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333042432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714410:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828023.714411:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714414:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714417:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714420:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714429:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828023.714444:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333042560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714457:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.714468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714473:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714478:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714482:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828023.714499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333042688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828023.714509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714512:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714518:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714523:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828023.714535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333042816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714541:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.714545:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714548:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714550:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714555:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828023.714576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333042944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828023.714619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714627:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714633:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714641:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828023.714655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333043072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714668:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.714673:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714676:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714679:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714681:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828023.714698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333043200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.714711:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.714718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.714721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.714727:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9834 00000100:00080000:1.0:1787828023.714731:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.714737:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828023.714744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828023.714757:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333043328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828024.098353:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828025.122389:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828026.146439:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828028.194481:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828028.834414:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.834435:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828028.834442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.834462:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.834501:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.834518:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828028.834626:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333043456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.834666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.834673:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828028.834676:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.834684:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.834691:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.834700:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.834718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.834738:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333043584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.834761:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.834766:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.834768:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.834778:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.834795:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.834803:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.834815:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.834844:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333043712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.834859:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.834865:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828028.834867:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.834873:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.834879:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.834890:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.834913:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828028.834936:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333043840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.834957:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.834983:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.834993:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.835011:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.835020:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.835028:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.835045:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828028.835074:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333043968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.835149:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.835156:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828028.835160:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.835179:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.835190:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.835198:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.835209:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.835286:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333044096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.835306:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.835310:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.835314:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.835331:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.835348:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.835363:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.835373:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.835423:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333044224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.835517:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.835526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.835532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.835549:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9839 00000100:00080000:3.0:1787828028.835581:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.835592:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828028.835625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828028.835662:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333044352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828029.218675:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828030.242491:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828031.266440:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828032.290522:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828033.954308:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954317:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828033.954321:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954329:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954339:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954346:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828033.954387:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333044480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954413:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954416:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828033.954418:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954421:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954424:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954427:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954436:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828033.954449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333044608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954472:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954475:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828033.954476:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954481:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828033.954496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333044736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828033.954506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954508:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954512:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954514:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828033.954542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333044864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828033.954554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954557:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954560:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828033.954579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333044992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828033.954609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954615:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954622:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954631:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828033.954644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333045120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954654:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828033.954655:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954658:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954661:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954666:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828033.954684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333045248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828033.954705:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828033.954708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828033.954711:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828033.954720:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9844 00000100:00080000:1.0:1787828033.954725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828033.954730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828033.954735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828033.954757:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333045376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828034.338554:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828035.362478:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828036.386537:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828038.434512:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828039.074668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333045504:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828039.074750:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.074813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de0700 x1874663333045504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.074851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333045504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.074871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.074888:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.074895:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333045504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828039.074928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.074962:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.074983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de2a00 x1874663333045632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333045632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.075121:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.075171:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075200:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de1180 x1874663333045760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075283:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333045760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075315:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.075351:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.075378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075392:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de1880 x1874663333045888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075425:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333045888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075440:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075444:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333045888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.075452:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.075478:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075545:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de3800 x1874663333046016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075565:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333046016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075584:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075589:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.075612:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.075635:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075646:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de0e00 x1874663333046144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075658:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333046144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075668:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075672:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.075678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.075694:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075702:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de0a80 x1874663333046272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075716:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333046272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075729:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075738:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.075747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.075776:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828039.075793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9057d0de0000 x1874663333046400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.075806:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333046400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.075814:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.075817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828039.075820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333046400:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828039.458373:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828040.482468:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828041.506456:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828042.530487:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828044.194442:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333046528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828044.194551:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.194593:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.194621:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828044.194639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828044.194654:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.194666:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.194681:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047424:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828044.578529:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828045.602506:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828046.626482:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828047.266651:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787828047.266730:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828047.266757:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266819:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266831:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266840:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266851:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266866:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266876:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828047.266886:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828047.266917:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de1500 x1874663333046784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.266956:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333046784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.266971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.266978:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.266984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828047.266993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de1c00 x1874663333046912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333046912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267033:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267037:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828047.267044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de0000 x1874663333047040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333047040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267107:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828047.267120:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de0a80 x1874663333047168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333047168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267174:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267180:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828047.267189:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de0e00 x1874663333047296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267207:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333047296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267225:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267230:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828047.267239:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de3100 x1874663333046528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267291:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333046528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267309:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267314:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333046528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828047.267323:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de1f80 x1874663333046656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267340:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333046656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267357:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267362:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333046656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828047.267402:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828047] req@ffff9057d0de3800 x1874663333047424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828047.267446:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333047424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828047.267459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828047.267464:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828047.267469:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333047424:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828048.674502:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828049.314461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.314476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828049.314483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.314499:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.314518:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828049.314533:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828049.314602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333047552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828049.314662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.314677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828049.314680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.314689:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.314699:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828049.314708:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828049.314727:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828049.314758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333047680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.314766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333047552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828049.314843:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.314851:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828049.314856:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.314866:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.314875:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828049.314883:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828049.314896:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828049.314918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828049.314932:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333047808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828049.314955:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.314963:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828049.314967:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.314976:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.314986:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828049.314996:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828049.315009:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb223800 x1874663333047552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828049.315010:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828049.315046:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333047936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.315067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333047552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828049.315073:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.315092:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828049.315101:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.315122:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:1.0:1787828049.315125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828049.315132:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828049.315143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828049.315150:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828049.315164:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333047552:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828049.315166:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828049.315205:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333048064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.315290:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047680:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828049.315316:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.315326:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828049.315331:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.315343:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.315353:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828049.315358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828049.315369:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828049.315383:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828049.315412:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb222680 x1874663333047680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828049.315438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333048192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.315506:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333047680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.315526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.315534:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.315541:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828049.315556:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828049.315605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828049.315630:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb220e00 x1874663333047808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828049.315654:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333047808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.315670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.315677:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.315685:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828049.315700:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047936:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828049.315727:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.315737:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828049.315742:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828049.315753:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9860 00000800:00000100:1.0:1787828049.315764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828049.315765:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828049.315783:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828049.315796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828049.315803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb220a80 x1874663333047936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828049.315831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333048320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.315841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333047936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828049.315890:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828049.315904:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828049.315911:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828049.315925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828049.315927:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9860 00000100:00080000:3.0:1787828049.315944:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828049.315949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828049.315961:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828049.315970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333047936:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787828049.315978:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828049.316014:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333048448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828049.316029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828049.316090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828049.316118:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb221180 x1874663333048064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828049.316146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333048064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.316162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.316181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.316189:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828049.316201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828049.316292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828049.316318:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb220000 x1874663333048192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828049.316343:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333048192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.316372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.316379:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.316386:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828049.316399:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828049.316466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828049.316491:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb223b80 x1874663333048320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828049.316529:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333048320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.316570:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.316577:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.316585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828049.316618:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828049.316661:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828049.316684:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828049/real 1787828049] req@ffff9057fb221500 x1874663333048448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828049.316710:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333048448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828049.316729:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828049.316735:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828049.316741:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333048448:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828049.698657:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828050.722561:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828051.746458:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828052.770525:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828054.434801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333048576:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828054.434920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.434993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de1880 x1874663333048576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435039:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333048576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333048576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828054.435193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828054.435345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435419:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de1180 x1874663333048704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435453:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333048704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435481:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435485:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828054.435494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828054.435524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435539:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de2a00 x1874663333048832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435553:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333048832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435577:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435581:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828054.435587:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828054.435607:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435619:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de0700 x1874663333048960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435633:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333048960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435646:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435650:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333048960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828054.435656:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828054.435674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435685:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de3800 x1874663333049088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435698:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333049088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435708:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435711:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435715:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828054.435736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828054.435757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de1f80 x1874663333049216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435782:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333049216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435796:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828054.435806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828054.435824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435835:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de3100 x1874663333049344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333049344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435858:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435861:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828054.435871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828054.435889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828054.435900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828054/real 1787828054] req@ffff9057d0de0e00 x1874663333049472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828054.435914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333049472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828054.435923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828054.435927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828054.435931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333049472:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828054.818530:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828055.842479:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828056.866522:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828058.914507:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828059.554709:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333049600:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828059.554844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.554924:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de0a80 x1874663333049600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.554968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333049600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555012:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333049600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828059.555069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828059.555122:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555152:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de0000 x1874663333049728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333049728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828059.555326:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828059.555389:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555405:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de1c00 x1874663333049856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555421:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333049856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555437:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555441:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828059.555449:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828059.555472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555495:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de1500 x1874663333049984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555511:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333049984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555535:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555539:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333049984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828059.555546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828059.555569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de0e00 x1874663333050112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555604:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333050112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555614:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555617:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555620:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828059.555627:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828059.555647:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555658:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de3100 x1874663333050240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333050240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555685:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555689:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828059.555695:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828059.555714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de1f80 x1874663333050368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333050368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828059.555769:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828059.555789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828059.555799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828059/real 1787828059] req@ffff9057d0de3800 x1874663333050496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828059.555811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333050496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828059.555821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828059.555825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828059.555839:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333050496:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828059.938440:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828060.962525:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828061.986594:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828063.010558:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828064.674448:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.674461:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828064.674466:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.674480:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.674494:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.674504:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828064.674566:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333050624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.674605:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.674618:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828064.674621:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.674630:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.674639:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.674649:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.674665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828064.674678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333050624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828064.674719:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333050752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.674748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.674753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828064.674755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.674761:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.674769:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.674774:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.674782:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828064.674787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828064.674809:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333050880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.674824:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.674828:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828064.674831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.674836:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.674841:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.674846:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828064.674854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158d880 x1874663333050624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828064.674858:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828064.674883:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333051008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828064.674894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333050624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828064.674907:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.674923:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828064.674931:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828064.674937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828064.674939:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.674948:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828064.674949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828064.674960:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.674965:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828064.674981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333050624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828064.674983:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333051136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.675003:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.675008:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828064.675011:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.675017:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.675021:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828064.675029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828064.675030:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.675044:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828064.675065:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333051264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.675080:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828064.675088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828064.675091:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828064.675097:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.675105:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.675112:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828064.675120:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158d180 x1874663333050752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828064.675121:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.675135:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828064.675159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333050752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828064.675163:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333051392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828064.675199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828064.675206:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828064.675212:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828064.675216:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828064.675269:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828064.675278:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828064.675291:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050880:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828064.675292:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9875 00000100:00080000:0.0:1787828064.675304:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.675344:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828064.675355:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828064.675366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828064.675383:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333051520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828064.675394:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158ce00 x1874663333050880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675436:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333050880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675456:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675463:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675471:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333050880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828064.675484:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828064.675527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828064.675552:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158c700 x1874663333051008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675577:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333051008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675594:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675601:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675608:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828064.675616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828064.675645:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828064.675663:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158ea00 x1874663333051136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333051136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675704:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675711:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675718:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828064.675754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828064.675784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828064.675806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158fb80 x1874663333051264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333051264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675836:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828064.675847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828064.675865:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828064.675876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158ed80 x1874663333051392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333051392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828064.675913:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828064.675931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828064.675941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff9057f158d500 x1874663333051520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.675955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333051520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.675964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.675967:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828064.675972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333051520:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828065.058552:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828066.082541:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828067.106486:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828069.154465:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828069.794389:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794402:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828069.794407:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.794421:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.794435:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828069.794444:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828069.794504:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333051648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828069.794539:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794544:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828069.794546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.794552:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.794558:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828069.794566:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828069.794579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828069.794589:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333051648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828069.794604:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333051776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828069.794623:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794627:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828069.794629:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.794633:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.794647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828069.794651:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828069.794658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828069.794659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828069.794683:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333051904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828069.794698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794704:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828069.794707:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158c000 x1874663333051648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828069.794709:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.794724:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.794733:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828069.794736:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333051648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828069.794743:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828069.794774:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828069.794775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828069.794791:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.794798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333051648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828069.794808:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333052032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828069.794823:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794828:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828069.794831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828069.794837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051776:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828069.794843:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.794851:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828069.794857:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828069.794865:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828069.794889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828069.794891:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333052160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828069.794913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158dc00 x1874663333051776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828069.794915:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.794949:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828069.794955:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828069.794960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333051776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828069.794967:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:1.0:1787828069.794973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828069.794977:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828069.794980:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.794986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051776:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828069.794991:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828069.794998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051904:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828069.795002:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828069.795024:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333052288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828069.795057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828069.795065:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.795072:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828069.795075:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.795085:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9880 00000100:00080000:0.0:1787828069.795095:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828069.795119:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158e300 x1874663333051904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828069.795125:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828069.795139:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828069.795143:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333051904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795161:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333051904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828069.795167:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333052416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828069.795172:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052032:192.168.203.156@tcp:8:kworker.0 00000100:00080000:0.0:1787828069.795215:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828069.795221:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828069.795225:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828069.795232:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9880 00000800:00000100:1.0:1787828069.795239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828069.795244:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828069.795256:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828069.795264:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828069.795368:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333052544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828069.795389:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158f100 x1874663333052032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.795412:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333052032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795431:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795436:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828069.795461:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828069.795487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828069.795498:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158ca80 x1874663333052160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.795515:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333052160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795525:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795527:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828069.795532:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828069.795547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828069.795555:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158df80 x1874663333052288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.795564:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333052288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795571:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795573:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795575:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828069.795599:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828069.795621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828069.795632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158e680 x1874663333052416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.795646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333052416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795664:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795668:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795671:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828069.795678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828069.795696:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828069.795723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff9057f158f800 x1874663333052544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.795733:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333052544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.795740:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.795742:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828069.795744:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333052544:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828070.178525:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828071.202509:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828072.226437:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828073.250464:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828074.914372:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.914386:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828074.914394:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.914410:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.914427:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914439:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828074.914504:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333052672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.914560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.914567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828074.914572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.914582:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.914591:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914601:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.914617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828074.914619:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333052672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828074.914648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333052800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.914666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.914671:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828074.914674:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.914680:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.914685:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914689:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.914696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828074.914710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333052928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.914733:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.914738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828074.914740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828074.914750:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828074.914758:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.914765:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914771:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.914779:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828074.914794:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333053056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.914806:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.914812:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828074.914814:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.914826:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00100000:1.0:1787828074.914828:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158f800 x1874663333052672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828074.914831:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914842:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.914854:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828074.914874:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333052672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.914893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828074.914895:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333053184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828074.914909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828074.914923:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828074.914925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333052672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828074.914935:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828074.914941:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.914949:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.914954:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.914959:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.914964:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828074.914975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828074.914982:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333053312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.914999:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.915005:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828074.915007:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.915013:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9885 00000800:00000100:1.0:1787828074.915021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828074.915022:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.915033:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828074.915040:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828074.915050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158e680 x1874663333052800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828074.915059:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333053440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828074.915096:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333052800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828074.915107:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828074.915110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828074.915117:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828074.915124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828074.915125:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828074.915139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052800:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828074.915141:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9885 00000100:00080000:0.0:1787828074.915151:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.915158:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915164:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052928:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828074.915168:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828074.915194:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333053568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828074.915208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915293:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158df80 x1874663333052928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915317:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333052928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915337:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915343:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333052928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828074.915362:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828074.915399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915426:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158ca80 x1874663333053056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915446:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333053056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915457:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915461:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915465:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828074.915471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828074.915491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915501:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158f100 x1874663333053184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915515:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333053184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915525:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915528:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915532:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828074.915555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828074.915575:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915586:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158e300 x1874663333053312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915600:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333053312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915614:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915618:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828074.915624:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828074.915643:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915653:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158dc00 x1874663333053440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915666:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333053440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915679:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828074.915689:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828074.915707:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828074.915717:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff9057f158c000 x1874663333053568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.915729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333053568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.915739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.915743:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828074.915748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333053568:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828075.298431:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828076.322435:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828077.346553:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828079.394515:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828080.034479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828080.034500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034515:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034541:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034550:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828080.034603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333053696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034644:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828080.034651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034658:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034662:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034667:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828080.034679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828080.034693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333053824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034710:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828080.034712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034716:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034720:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034725:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828080.034730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828080.034743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333053952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034755:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034760:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828080.034763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034767:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034771:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034781:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828080.034785:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333053696:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828080.034788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828080.034807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333054080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034822:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034827:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828080.034829:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034834:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034838:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034842:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828080.034848:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828080.034863:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333054208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034875:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034880:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828080.034883:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034887:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034901:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828080.034909:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828080.034911:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828080.034923:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828080.034941:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333054336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828080.034954:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828080.034959:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828080.034961:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.034966:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:3.0:1787828080.034972:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828080.034982:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828080.034983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb221500 x1874663333053696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828080.034993:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828080.035018:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333054464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828080.035027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333053696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828080.035049:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828080.035058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828080.035058:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828080.035066:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828080.035082:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9890 00000100:00080000:1.0:1787828080.035083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828080.035087:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828080.035094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333053696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828080.035097:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828080.035111:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828080.035118:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828080.035129:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333054592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828080.035159:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb223b80 x1874663333053824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035279:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333053824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035300:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035305:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.035314:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828080.035348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035363:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb220000 x1874663333053952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035386:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333053952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035400:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035404:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333053952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.035437:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.035458:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035469:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb221180 x1874663333054080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035483:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333054080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035496:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035499:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828080.035506:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.035525:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035535:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb220a80 x1874663333054208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035569:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333054208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035578:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035581:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828080.035591:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828080.035610:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035620:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb220e00 x1874663333054336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035632:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333054336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035645:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035648:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.035655:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828080.035673:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb222680 x1874663333054464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035696:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333054464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035720:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035734:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.035745:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.035782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828080.035804:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9057fb223800 x1874663333054592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.035829:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333054592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.035840:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.035844:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828080.035848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333054592:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828080.418412:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828081.442492:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828082.466484:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828083.490531:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828085.154317:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154328:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828085.154333:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154344:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154356:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154364:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828085.154415:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333054720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828085.154458:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828085.154469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828085.154528:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333054848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828085.154543:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333054720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828085.154563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828085.154575:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154582:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154587:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154592:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828085.154618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333054976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828085.154639:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154643:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828085.154645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154650:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154653:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828085.154669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828085.154681:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333055104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828085.154694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154698:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828085.154700:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154704:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154716:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154720:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828085.154740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333055232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828085.154749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb221c00 x1874663333054720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828085.154755:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828085.154795:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154807:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00100000:1.0:1787828085.154809:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333054720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828085.154812:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154822:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154833:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828085.154851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828085.154854:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333055360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.154869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828085.154872:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154880:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828085.154887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333054720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828085.154887:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154900:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154905:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828085.154910:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.154917:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828085.154935:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333055488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828085.154938:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828085.154969:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828085.154976:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828085.154980:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828085.154988:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9895 00000100:00080000:3.0:1787828085.154995:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828085.154997:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828085.155008:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828085.155016:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828085.155031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb222a00 x1874663333054848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828085.155037:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333055616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828085.155108:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333054848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155133:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828085.155152:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828085.155284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155309:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb222d80 x1874663333054976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155332:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333054976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155354:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155361:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333054976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828085.155372:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828085.155410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155431:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb223100 x1874663333055104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155453:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333055104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155474:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155480:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828085.155491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828085.155527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155546:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb221880 x1874663333055232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155575:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333055232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155596:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155602:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828085.155612:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828085.155655:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155674:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb220700 x1874663333055360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155695:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333055360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155726:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155732:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828085.155766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828085.155820:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb220380 x1874663333055488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333055488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155858:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155861:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828085.155871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828085.155889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828085.155899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9057fb222300 x1874663333055616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.155911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333055616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.155920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828085.155923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828085.155928:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333055616:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828085.538459:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828086.562519:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828087.586445:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828089.634486:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828090.274377:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828090.274393:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.274404:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.274417:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828090.274425:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828090.274472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333055744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828090.274514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828090.274520:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.274525:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.274528:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828090.274532:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828090.274542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828090.274557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333055872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828090.274569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828090.274576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.274581:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.274586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828090.274593:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828090.274603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333055744:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828090.274605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828090.274674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333056000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828090.274709:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274714:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828090.274717:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.274723:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.274729:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828090.274737:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828090.274738:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828090.274752:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828090.274771:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333056128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828090.274783:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274787:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828090.274790:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.274794:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.274799:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828090.274804:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb222300 x1874663333055744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828090.274810:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828090.274821:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828090.274856:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333056256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828090.274861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333055744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828090.274889:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.274895:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828090.274896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828090.274900:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828090.274906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828090.274913:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00100000:1.0:1787828090.274920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333055744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828090.274922:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828090.274935:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828090.274942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055872:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828090.274956:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828090.274978:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333056384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828090.274998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828090.275009:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828090.275026:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828090.275046:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.275054:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.275061:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828090.275063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb220380 x1874663333055872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828090.275070:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828090.275083:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828090.275111:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333056512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828090.275117:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333055872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828090.275199:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828090.275202:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275213:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333055872:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828090.275214:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828090.275223:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828090.275313:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9901 00000100:00080000:3.0:1787828090.275340:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828090.275347:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275357:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056000:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828090.275358:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828090.275404:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333056640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828090.275413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275433:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb220700 x1874663333056000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275450:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333056000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275465:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275468:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828090.275475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828090.275493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275504:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb221880 x1874663333056128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275516:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333056128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275524:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275527:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275530:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828090.275538:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828090.275565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275584:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb223100 x1874663333056256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275618:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333056256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275632:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275636:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828090.275666:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828090.275694:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275708:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb222d80 x1874663333056384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275722:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333056384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275732:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275753:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275757:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828090.275764:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828090.275784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb222a00 x1874663333056512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275813:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333056512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828090.275859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828090.275878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828090.275888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828090/real 1787828090] req@ffff9057fb221c00 x1874663333056640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828090.275901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333056640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828090.275910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828090.275913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828090.275916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333056640:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828090.658483:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828091.682397:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828092.706461:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828093.730661:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828095.394370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394385:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828095.394391:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394407:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394424:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394435:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828095.394496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333056768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828095.394567:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394574:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828095.394578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394587:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394618:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.394638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828095.394644:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333056768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828095.394664:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333056896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828095.394682:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828095.394689:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394695:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.394713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828095.394728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333057024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828095.394741:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828095.394748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394752:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394756:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394764:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828095.394769:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828095.394773:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828095.394789:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333057152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828095.394802:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394806:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828095.394809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394815:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394819:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394824:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.394831:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828095.394845:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333057280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828095.394851:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158d500 x1874663333056768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828095.394876:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394885:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828095.394887:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.394893:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394896:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.394901:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.394906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828095.394915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333056768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828095.394922:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333057408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828095.394955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828095.394958:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.394969:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828095.394970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828095.394973:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828095.394983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333056768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828095.394987:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.394997:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828095.395001:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.395006:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828095.395018:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333057536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828095.395044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056896:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828095.395063:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828095.395069:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828095.395072:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828095.395079:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9906 00000100:00080000:0.0:1787828095.395136:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828095.395139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828095.395159:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828095.395168:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828095.395179:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158ed80 x1874663333056896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828095.395188:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333057664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828095.395285:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333056896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395306:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395311:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333056896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828095.395321:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828095.395353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395368:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158fb80 x1874663333057024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395382:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333057024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395395:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395399:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828095.395405:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828095.395425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395437:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158ea00 x1874663333057152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395450:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333057152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395462:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395465:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828095.395472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828095.395490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395501:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158c700 x1874663333057280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395514:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333057280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395538:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395544:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828095.395554:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828095.395604:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395626:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158ce00 x1874663333057408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395650:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333057408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395675:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828095.395704:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828095.395736:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395757:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158d180 x1874663333057536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333057536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395811:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828095.395823:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828095.395872:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828095.395886:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828095/real 1787828095] req@ffff9057f158d880 x1874663333057664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828095.395919:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333057664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828095.395929:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828095.395932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828095.395937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333057664:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828095.778478:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828096.802585:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828097.826549:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828099.874585:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828100.514406:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.514434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828100.514440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828100.514458:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.514473:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.514482:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828100.514530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333057792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828100.514569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.514574:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828100.514577:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828100.514593:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.514599:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.514605:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828100.514618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828100.514635:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333057920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828100.514650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.514655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828100.514659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828100.514668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333057792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828100.514669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.514684:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.514693:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828100.514705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828100.514732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333058048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828100.514752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.514761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828100.514766:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828100.514775:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9911 00000800:00000100:1.0:1787828100.514782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828100.514791:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.514806:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828100.514818:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828100.514847:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333058176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828100.514849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb223800 x1874663333057792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828100.514921:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.514927:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828100.514930:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828100.514939:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333057792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828100.514946:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.514960:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.514972:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828100.514974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.514998:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828100.514998:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828100.515012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333057792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828100.515025:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333058304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828100.515037:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.515042:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828100.515046:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828100.515054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057920:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828100.515088:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.515097:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.515104:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828100.515114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828100.515121:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828100.515134:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb222680 x1874663333057920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828100.515146:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333058432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828100.515153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333057920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828100.515202:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828100.515205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828100.515215:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828100.515256:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828100.515263:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.515269:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828100.515274:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828100.515281:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828100.515296:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333058560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828100.515310:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828100.515315:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828100.515318:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828100.515323:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9911 00000100:00080000:3.0:1787828100.515327:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828100.515334:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828100.515336:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828100.515349:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828100.515358:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333057920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828100.515366:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333058688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828100.515423:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828100.515453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515465:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb220e00 x1874663333058048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515475:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333058048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515481:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515484:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515487:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828100.515494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828100.515517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515530:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb220a80 x1874663333058176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515542:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333058176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515551:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515554:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515558:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828100.515566:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828100.515588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515601:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb221180 x1874663333058304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515616:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333058304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515622:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515624:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515627:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828100.515645:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828100.515660:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515668:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb220000 x1874663333058432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515676:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333058432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515681:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515683:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515684:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828100.515689:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828100.515701:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515707:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb223b80 x1874663333058560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515715:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333058560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515725:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828100.515734:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828100.515744:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828100.515750:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828100/real 1787828100] req@ffff9057fb221500 x1874663333058688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828100.515757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333058688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828100.515763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828100.515765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828100.515767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333058688:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828100.898552:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828101.922554:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828102.946554:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828103.970506:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828105.634305:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634317:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828105.634323:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634335:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634348:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634357:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828105.634400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333058816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.634432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828105.634439:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634443:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634447:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634451:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828105.634476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333058944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.634489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828105.634514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634519:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634525:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828105.634533:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333058816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828105.634539:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634559:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828105.634588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333059072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.634627:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828105.634636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634642:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634648:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634653:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828105.634677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333059200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.634689:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828105.634696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828105.634705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828105.634706:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634716:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634721:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828105.634743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333059328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.634757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828105.634764:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634772:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00100000:1.0:1787828105.634773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb221500 x1874663333058816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828105.634780:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634794:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828105.634821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333058816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828105.634833:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333059456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828105.634841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.634858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.634871:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333058816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828105.634873:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.634888:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828105.634890:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058944:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828105.634894:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.634908:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:3.0:1787828105.634922:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.634928:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828105.634937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828105.634942:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828105.634960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb223b80 x1874663333058944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828105.634970:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333059584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828105.634998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333058944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828105.634999:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.635015:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828105.635065:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828105.635077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828105.635079:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9916 00000100:00080000:1.0:1787828105.635091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828105.635091:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.635101:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635105:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333058944:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828105.635111:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828105.635130:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333059712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828105.635153:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828105.635185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635199:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb220000 x1874663333059072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635212:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333059072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635258:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635262:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828105.635269:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828105.635293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635311:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb221180 x1874663333059200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635333:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333059200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635352:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635358:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828105.635369:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828105.635401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635421:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb220a80 x1874663333059328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635441:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333059328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635460:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635466:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828105.635476:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828105.635509:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635525:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb220e00 x1874663333059456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635547:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333059456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635559:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635563:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828105.635569:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828105.635589:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635599:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb222680 x1874663333059584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635611:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333059584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635621:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635624:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828105.635645:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828105.635663:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828105.635673:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff9057fb223800 x1874663333059712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.635684:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333059712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.635717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.635721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828105.635725:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333059712:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828106.018541:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828107.042530:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828108.066633:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828110.114546:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828110.754496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.754508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828110.754514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.754527:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.754542:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.754555:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828110.754616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333059840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.754668:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.754676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828110.754680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.754689:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.754697:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.754719:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.754736:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828110.754754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333059840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828110.754762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333059968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.754783:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.754789:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828110.754792:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.754798:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.754804:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.754811:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.754820:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828110.754841:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333060096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.754859:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.754865:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828110.754868:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.754875:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.754881:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.754887:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.754896:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828110.754910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828110.754914:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333060224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.754933:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.754940:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828110.754944:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828110.754964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828110.754971:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.754978:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.755010:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.755021:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828110.755043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828110.755043:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333060352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.755069:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.755075:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828110.755079:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.755103:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00100000:1.0:1787828110.755107:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060352:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828110.755111:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.755119:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.755129:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828110.755150:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333060480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.755183:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.755190:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828110.755194:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.755202:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00080000:3.0:1787828110.755210:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.755260:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.755273:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828110.755298:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828110.755315:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333060608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.755336:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.755343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828110.755346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.755354:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9921 00000100:00100000:1.0:1787828110.755355:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060608:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828110.755361:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.755368:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828110.755377:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828110.755399:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333060736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828110.755451:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060736:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828111.138477:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828112.162519:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828113.186468:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828113.826772:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787828113.826843:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828113.826874:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828113.826941:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828113.826958:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828113.827012:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828113.827047:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828113.827050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb222a00 x1874663333059968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828113.827094:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828113.827116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333059968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.827132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.827148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000800:00000100:0.0:1787828113.827150:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828113.827162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333059968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1787828113.827176:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828113.827182:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb222d80 x1874663333060096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.827211:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333060096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.827275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.827282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.827290:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828113.827306:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb221c00 x1874663333059840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.827325:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333059840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.827335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.827340:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.827344:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333059840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828113.827378:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb222300 x1874663333060736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.829403:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333060736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.829422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.829444:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.829451:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828113.829463:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb220380 x1874663333060608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.829477:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333060608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.829484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.829487:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.829490:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828113.829495:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb220700 x1874663333060480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.829504:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333060480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.829511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.829516:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.829520:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333060480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828113.829529:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb221880 x1874663333060352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.829546:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333060352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.829557:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.829563:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.829569:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828113.829579:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828113] req@ffff9057fb223100 x1874663333060224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828113.829597:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333060224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828113.829605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828113.829609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828113.829615:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060224:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828114.210433:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828115.874372:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874386:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828115.874391:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.874404:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874417:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874438:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828115.874486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333060864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.874554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874559:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828115.874561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.874568:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874573:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874579:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828115.874619:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828115.874634:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333060864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828115.874638:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333060992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.874652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828115.874677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.874683:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874688:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874695:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828115.874703:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828115.874720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333061120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.874754:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828115.874758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828115.874758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828115.874761:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.874765:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874768:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874772:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828115.874795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828115.874807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333061248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828115.874808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb223100 x1874663333060864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828115.874820:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874824:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828115.874826:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828115.874829:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333060864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828115.874830:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874834:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874839:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828115.874842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828115.874843:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828115.874847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.874853:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333060864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828115.874855:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333061376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.874864:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874868:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828115.874870:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.874875:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00100000:1.0:1787828115.874879:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060992:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828115.874879:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874883:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828115.874889:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828115.874900:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333061504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828115.874905:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828115.874913:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874928:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828115.874930:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828115.874931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb221880 x1874663333060992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828115.874935:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.874939:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.874943:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828115.874947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333060992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828115.874949:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828115.874977:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.874981:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828115.874982:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333061632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828115.874986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333060992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.874994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061120:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828115.874995:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.874999:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828115.875002:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.875006:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9926 00000100:00080000:3.0:1787828115.875010:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.875015:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828115.875018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828115.875021:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828115.875033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb220700 x1874663333061120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828115.875034:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333061760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828115.875050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333061120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875065:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.875077:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828115.875104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828115.875119:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb220380 x1874663333061248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.875133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333061248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875157:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875161:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828115.875169:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828115.875205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828115.875312:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb222300 x1874663333061376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.875332:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333061376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875347:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875351:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333061376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828115.875393:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828115.875423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828115.875440:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb221c00 x1874663333061504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.875454:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333061504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875467:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875472:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.875480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828115.875507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828115.875522:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb222d80 x1874663333061632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.875538:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333061632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875549:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875553:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875558:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.875565:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828115.875591:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828115.875606:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff9057fb222a00 x1874663333061760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.875639:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333061760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.875650:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.875655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828115.875659:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333061760:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828116.258445:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828117.282622:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828118.306474:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828120.354505:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828120.994304:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994313:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828120.994317:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.994327:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994336:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994343:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828120.994379:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333061888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.994435:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828120.994446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.994454:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994461:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828120.994525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333062016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828120.994527:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333061888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828120.994563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828120.994572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.994578:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994592:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828120.994622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333062144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.994636:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828120.994648:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828120.994649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828120.994659:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994665:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994669:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828120.994706:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333062272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.994713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828120.994716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.994719:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994721:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994724:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828120.994737:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333062400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.994749:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994752:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828120.994754:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828120.994776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb223800 x1874663333061888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828120.994777:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994788:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994803:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994815:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828120.994827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333061888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828120.994835:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333062528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828120.994867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828120.994869:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994881:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828120.994882:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828120.994890:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828120.994898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333061888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828120.994906:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.994960:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.994964:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.994969:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828120.994974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828120.994982:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333062656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.994993:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.994996:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828120.994998:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828120.995003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828120.995026:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9931 00000100:00080000:3.0:1787828120.995035:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.995041:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828120.995048:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828120.995053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb222680 x1874663333062016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828120.995070:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333062784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828120.995081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333062016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995112:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.995129:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828120.995157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb220e00 x1874663333062144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995192:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333062144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995227:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995230:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.995234:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828120.995247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995288:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb220a80 x1874663333062272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333062272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995307:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995309:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828120.995314:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828120.995334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995346:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb221180 x1874663333062400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995361:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333062400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995402:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995406:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333062400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828120.995415:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828120.995445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995467:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb220000 x1874663333062528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995491:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333062528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995515:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995522:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.995559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828120.995589:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995603:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb223b80 x1874663333062656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995618:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333062656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995628:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995632:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995637:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.995644:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828120.995664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828120.995675:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff9057fb221500 x1874663333062784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.995704:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333062784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.995710:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.995712:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828120.995714:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333062784:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828121.378523:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828122.402570:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828123.426580:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828124.450533:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828126.114420:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114439:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828126.114447:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.114462:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114478:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.114490:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828126.114549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333062912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.114593:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.114601:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.114607:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114611:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.114616:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828126.114629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828126.114645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333063040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.114668:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.114675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.114680:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114685:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.114689:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828126.114696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828126.114703:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333062912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828126.114713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333063168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.114729:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114736:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.114738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.114745:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114750:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.114756:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828126.114764:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828126.114776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333063296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.114798:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114803:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.114805:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.114809:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114817:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828126.114817:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828126.114834:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828126.114842:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828126.114859:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333063424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.114872:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.114877:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.114903:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828126.114907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158d880 x1874663333062912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828126.114914:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.114945:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.114965:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828126.114977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333062912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828126.114979:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828126.115002:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115007:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333062912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828126.115023:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333063552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828126.115045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063040:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828126.115064:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.115071:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.115075:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.115084:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.115091:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.115099:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828126.115104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828126.115110:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828126.115130:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333063680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828126.115140:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158d180 x1874663333063040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828126.115207:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.115213:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.115216:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.115280:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9936 00000100:00080000:0.0:1787828126.115285:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.115291:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115300:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333063040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828126.115304:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828126.115330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828126.115339:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333063808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828126.115341:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115351:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.115401:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828126.115443:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115456:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158ce00 x1874663333063168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115467:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333063168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115475:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115478:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115481:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.115487:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.115503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115520:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158c700 x1874663333063296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115538:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333063296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115547:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115550:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828126.115555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.115571:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158ea00 x1874663333063424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115590:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333063424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115597:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115600:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115602:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333063424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828126.115607:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828126.115622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115630:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158fb80 x1874663333063552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115640:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333063552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115652:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115658:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.115663:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828126.115677:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115685:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158ed80 x1874663333063680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115694:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333063680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115701:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115704:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.115712:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.115725:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828126.115733:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9057f158d500 x1874663333063808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.115741:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333063808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.115748:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.115751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828126.115754:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333063808:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828126.498624:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828127.522433:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828128.546568:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828130.594411:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828131.234381:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.234397:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828131.234404:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.234419:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234436:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234447:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828131.234510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333063936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.234558:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.234564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828131.234567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.234573:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234577:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234583:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.234597:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828131.234616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333064064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.234632:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828131.234647:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333063936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828131.234657:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828131.234663:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.234669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234674:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234679:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.234687:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828131.234705:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333064192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.234718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.234723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828131.234725:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.234730:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234735:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.234747:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828131.234768:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828131.234794:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333064320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.234811:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.234815:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828131.234818:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.234825:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234831:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234837:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.234871:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828131.234878:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb221500 x1874663333063936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828131.234896:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333064448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.234919:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.234925:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828131.234929:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828131.234932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333063936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828131.234940:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.234952:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.234964:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828131.234965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828131.234974:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828131.234978:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.234990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333063936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828131.235000:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333064576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828131.235024:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828131.235031:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828131.235035:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828131.235039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064064:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828131.235050:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.235061:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.235071:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.235083:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828131.235086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb223b80 x1874663333064064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828131.235131:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333064704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828131.235135:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333064064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828131.235186:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828131.235191:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828131.235199:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828131.235204:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064064:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828131.235206:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828131.235274:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9941 00000100:00080000:3.0:1787828131.235284:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828131.235292:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828131.235301:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828131.235319:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333064832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828131.235335:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828131.235377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235394:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb220000 x1874663333064192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235410:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333064192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235431:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235438:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828131.235450:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828131.235488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235511:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb221180 x1874663333064320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235537:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333064320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235549:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235553:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235557:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828131.235565:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828131.235593:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235607:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb220a80 x1874663333064448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235631:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333064448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235644:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235648:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333064448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828131.235654:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828131.235675:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235686:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb220e00 x1874663333064576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235700:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333064576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235710:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235713:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235716:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828131.235742:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828131.235761:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb222680 x1874663333064704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333064704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235829:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828131.235841:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828131.235908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828131.235930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828131/real 1787828131] req@ffff9057fb223800 x1874663333064832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828131.235946:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333064832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828131.235957:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828131.235961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828131.235965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333064832:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828131.618494:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828132.642515:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828133.666433:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828134.690428:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828136.354407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354423:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828136.354429:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354445:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354462:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354473:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828136.354534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333064960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828136.354584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354590:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354594:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354599:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354612:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828136.354627:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333065088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354654:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828136.354658:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354662:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354666:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354677:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828136.354693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333065216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828136.354717:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333064960:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828136.354719:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354735:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354740:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354745:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354752:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828136.354769:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333065344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354782:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354787:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828136.354790:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354795:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354803:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354806:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354837:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828136.354857:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333065472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354877:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354882:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828136.354884:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354894:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9947 00000800:00000100:1.0:1787828136.354896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828136.354902:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354911:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354919:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828136.354948:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333065600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.354960:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.354965:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828136.354967:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828136.354973:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.354977:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.354983:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.354999:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828136.355002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb222a00 x1874663333064960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828136.355017:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333065728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828136.355075:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828136.355080:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828136.355084:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828136.355090:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333064960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828136.355093:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9947 00000100:00080000:3.0:1787828136.355106:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828136.355121:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828136.355142:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828136.355148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355163:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828136.355174:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333065856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828136.355181:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333064960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828136.355214:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828136.355310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355334:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb222d80 x1874663333065088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355353:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333065088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355369:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355373:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828136.355410:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828136.355434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355447:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb221c00 x1874663333065216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355461:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333065216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355474:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355478:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828136.355485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828136.355507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355522:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb222300 x1874663333065344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355546:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333065344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355561:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355567:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355573:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828136.355586:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828136.355627:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355650:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb220380 x1874663333065472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355676:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333065472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333065472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828136.355717:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828136.355748:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355764:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb220700 x1874663333065600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355788:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333065600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828136.355843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828136.355867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb221880 x1874663333065728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333065728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355912:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828136.355919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828136.355939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828136.355949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828136/real 1787828136] req@ffff9057fb223100 x1874663333065856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828136.355963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333065856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828136.355982:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828136.355986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828136.355990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333065856:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828136.738591:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828137.762561:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828138.786413:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828140.834411:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828141.474364:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474377:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828141.474382:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474393:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474413:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474422:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828141.474471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333065984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474504:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828141.474509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474514:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474517:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474520:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828141.474540:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333066112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828141.474554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474558:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474560:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828141.474585:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333066240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828141.474598:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474604:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474608:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828141.474608:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333065984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828141.474617:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474623:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828141.474640:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333066368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474647:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474651:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828141.474653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474656:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474659:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474662:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828141.474675:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333066496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474687:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828141.474689:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474692:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474695:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474698:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828141.474705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828141.474715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333066624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474729:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474733:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828141.474734:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828141.474737:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474740:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474742:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828141.474746:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828141.474756:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333066752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828141.474764:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828141.474767:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828141.474771:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828141.474787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158c000 x1874663333065984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828141.474794:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9952 00000100:00080000:0.0:1787828141.474815:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828141.474820:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828141.474827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333065984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828141.474832:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828141.474845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828141.474851:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333066880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828141.474852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.474863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333065984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828141.474878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828141.474903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.474915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158dc00 x1874663333066112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.474924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333066112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.474931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.474933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.474937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828141.474963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828141.474977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.474985:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158e300 x1874663333066240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.474994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333066240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475002:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828141.475009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828141.475021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.475028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158f100 x1874663333066368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.475037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333066368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475043:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828141.475051:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828141.475063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.475070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158ca80 x1874663333066496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.475078:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333066496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333066496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828141.475093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828141.475105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.475111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158df80 x1874663333066624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.475119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333066624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475130:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828141.475134:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828141.475146:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.475152:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158e680 x1874663333066752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.475161:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333066752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475170:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828141.475177:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828141.475189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828141.475195:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828141/real 1787828141] req@ffff9057f158f800 x1874663333066880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828141.475203:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333066880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828141.475208:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828141.475211:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828141.475213:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333066880:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828141.858425:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828142.882491:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828143.906585:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828144.930401:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828146.594340:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594351:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828146.594356:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594368:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.594379:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.594387:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828146.594435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333067008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.594469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828146.594475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.594483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.594486:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.594496:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828146.594518:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333067136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.594529:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.594546:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594552:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.594557:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.594561:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.594567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828146.594578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333067264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.594604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828146.594609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.594627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828146.594630:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333067008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828146.594633:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.594648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828146.594665:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333067392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.594675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594680:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.594682:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.594690:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.594694:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.594701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828146.594713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333067520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.594724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.594727:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828146.594729:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.594733:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9957 00000800:00000100:1.0:1787828146.594786:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.594852:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158f800 x1874663333067008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.594879:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333067008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.594896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.594903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.594909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333067008:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828146.594920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067136:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828146.594949:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.594960:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.594968:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828146.594977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828146.595000:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333067648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.595014:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828146.595018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158e680 x1874663333067136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828146.595020:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.595027:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.595054:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.595067:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828146.595073:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333067136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828146.595083:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828146.595103:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828146.595111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828146.595132:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333067776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828146.595140:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.595161:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067136:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828146.595184:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.595189:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.595193:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.595200:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9957 00000100:00080000:0.0:1787828146.595207:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.595214:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595235:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067264:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828146.595251:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828146.595272:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333067904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828146.595397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595417:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158df80 x1874663333067264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.595434:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333067264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.595446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.595450:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.595455:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828146.595464:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828146.595488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595500:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158ca80 x1874663333067392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.595532:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333067392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.595540:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.595545:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.595551:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828146.595562:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828146.595614:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595635:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158f100 x1874663333067520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.595659:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333067520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.595676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.595682:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.595689:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333067520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828146.595700:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828146.595780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158e300 x1874663333067648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.595855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333067648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.595871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.595877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.595883:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828146.595895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828146.595935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.595972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158dc00 x1874663333067776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.595994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333067776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.596008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.596014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.596020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828146.596054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828146.596083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828146.596096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff9057f158c000 x1874663333067904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828146.596109:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333067904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828146.596146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828146.596150:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828146.596154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333067904:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828146.978451:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828148.002444:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828149.026479:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828151.074615:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828151.714410:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.714426:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828151.714432:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.714448:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.714466:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.714477:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828151.714537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333068032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.714617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.714622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828151.714625:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.714632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.714637:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.714642:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.714657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828151.714675:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333068160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.714688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.714693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828151.714695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.714726:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.714752:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.714758:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.714766:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828151.714782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333068288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.714816:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828151.714817:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333068032:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828151.714828:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828151.714835:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.714843:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.714848:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.714853:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.714860:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828151.714895:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333068416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.714907:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.714912:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828151.714914:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.714918:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.714962:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.714967:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.714973:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828151.714987:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333068544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828151.715014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828151.715037:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.715044:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828151.715047:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.715056:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.715062:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.715067:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.715092:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828151.715124:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158d500 x1874663333068032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828151.715125:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333068672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.715190:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.715195:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828151.715198:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.715207:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00100000:1.0:1787828151.715208:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333068032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828151.715211:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828151.715225:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828151.715249:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828151.715269:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828151.715286:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333068800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828151.715299:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828151.715304:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828151.715307:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828151.715312:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9962 00000100:00080000:0.0:1787828151.715323:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828151.715333:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828151.715335:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828151.715349:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828151.715350:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333068032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828151.715369:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333068928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828151.715374:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828151.715458:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.715480:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158ed80 x1874663333068160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.715497:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333068160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.715508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.715513:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.715521:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828151.715588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828151.715628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.715651:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158fb80 x1874663333068288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.715676:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333068288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.715692:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.715699:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.715731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828151.715740:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828151.715791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.715808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158ea00 x1874663333068416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.715823:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333068416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.715834:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.715840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.715847:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828151.715860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828151.715912:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.715935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158c700 x1874663333068544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.715959:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333068544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.715973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.715976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.715980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333068544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828151.716006:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828151.716035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.716053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158ce00 x1874663333068672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.716068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333068672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.716094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.716098:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.716101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828151.716107:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828151.716128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.716138:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158d180 x1874663333068800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.716151:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333068800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.716161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.716164:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.716168:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828151.716174:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828151.716192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828151.716202:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828151] req@ffff9057f158d880 x1874663333068928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828151.716213:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333068928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828151.716222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828151.716225:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828151.716231:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333068928:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828152.098482:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828153.122571:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828154.146504:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828155.170461:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828156.834353:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834368:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828156.834373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834387:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834403:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834414:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828156.834474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333069056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828156.834535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828156.834541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834545:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834549:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834553:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828156.834577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333069184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828156.834592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828156.834597:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834603:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00100000:1.0:1787828156.834603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333069056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828156.834625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834638:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828156.834660:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333069312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828156.834691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828156.834700:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834705:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834727:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834732:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834738:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828156.834748:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828156.834754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333069440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828156.834766:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828156.834773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834778:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834784:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834791:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834797:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828156.834808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158d880 x1874663333069056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828156.834811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333069568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828156.834842:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834846:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828156.834847:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834851:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834855:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828156.834859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333069056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828156.834863:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834869:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828156.834880:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333069696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828156.834883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828156.834892:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828156.834894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828156.834898:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828156.834901:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828156.834902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333069056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828156.834909:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834912:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834915:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834918:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828156.834925:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333069824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828156.834934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069184:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828156.834946:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828156.834950:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828156.834951:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828156.834955:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9967 00000100:00080000:0.0:1787828156.834960:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828156.834967:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828156.834967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828156.834975:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828156.834985:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333069952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828156.834987:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158d180 x1874663333069184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333069184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835020:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835026:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828156.835032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828156.835047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158ce00 x1874663333069312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835074:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333069312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835081:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828156.835088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828156.835103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158c700 x1874663333069440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333069440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828156.835131:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828156.835143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835148:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158ea00 x1874663333069568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835156:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333069568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835164:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835165:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333069568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828156.835169:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828156.835180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835186:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158fb80 x1874663333069696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835200:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333069696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835207:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835209:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828156.835251:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828156.835265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835273:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158ed80 x1874663333069824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835281:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333069824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835289:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828156.835296:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828156.835306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828156.835312:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff9057f158d500 x1874663333069952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828156.835320:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333069952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828156.835331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828156.835333:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828156.835335:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333069952:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828157.218424:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828158.242517:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828159.266499:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828161.314382:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828161.954375:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828161.954393:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.954417:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954431:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954441:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828161.954501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333070080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828161.954559:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.954565:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954569:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954574:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.954586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828161.954600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333070208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954618:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828161.954620:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.954623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954661:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.954668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828161.954683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333070336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828161.954704:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828161.954713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333070080:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828161.954715:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954732:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.954739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828161.954755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333070464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954769:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954774:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828161.954776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.954781:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954797:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954802:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.954808:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828161.954814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828161.954832:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333070592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954846:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954850:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828161.954852:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.954859:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954865:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954871:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828161.954879:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158c000 x1874663333070080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828161.954880:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828161.954904:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333070720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828161.954920:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828161.954925:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828161.954928:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828161.954935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333070080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828161.954938:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00080000:0.0:1787828161.954947:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.954959:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.954972:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828161.954985:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828161.954993:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333070848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828161.955003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828161.955028:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828161.955033:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333070080:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828161.955035:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828161.955043:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828161.955053:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9972 00000100:00100000:1.0:1787828161.955057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070208:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828161.955061:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828161.955076:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828161.955083:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828161.955101:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333070976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828161.955117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955151:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158dc00 x1874663333070208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955169:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333070208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955185:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955190:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.955321:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828161.955349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955364:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158e300 x1874663333070336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955380:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333070336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955394:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955399:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.955405:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828161.955425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955436:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158f100 x1874663333070464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955450:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333070464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955463:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955466:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828161.955472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828161.955492:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955503:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158ca80 x1874663333070592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955517:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333070592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955530:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955533:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333070592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828161.955540:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828161.955558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955568:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158df80 x1874663333070720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955581:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333070720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955594:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955598:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.955604:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828161.955623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955669:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158e680 x1874663333070848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955694:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333070848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955705:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955709:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955713:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.955721:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828161.955748:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828161.955761:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff9057f158f800 x1874663333070976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.955775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333070976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.955784:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.955788:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828161.955791:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333070976:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828162.338554:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828163.362546:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828164.386686:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828165.410725:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828167.074355:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828167.074372:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074385:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074402:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074413:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828167.074473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333071104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828167.074526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828167.074532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074537:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074541:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074545:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.074557:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828167.074580:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333071232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828167.074591:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828167.074613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074617:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074638:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074643:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.074649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828167.074660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333071104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828167.074663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333071360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828167.074687:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074690:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828167.074691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074695:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074700:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074703:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.074710:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828167.074721:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333071488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828167.074735:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074741:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787828167.074745:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828167.074748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074759:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074766:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074772:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.074781:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828167.074800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158f800 x1874663333071104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828167.074801:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333071616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828167.074856:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074860:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828167.074861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074871:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00100000:1.0:1787828167.074872:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333071104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828167.074880:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828167.074897:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828167.074903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828167.074906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828167.074911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.074918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333071104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828167.074923:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333071744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828167.074938:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071232:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828167.074960:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828167.074974:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828167.074977:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.074984:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.074988:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828167.074995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828167.075002:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.075011:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828167.075013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158e680 x1874663333071232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333071232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828167.075049:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333071872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828167.075126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828167.075130:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828167.075136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828167.075141:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828167.075147:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828167.075166:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9977 00000100:00080000:0.0:1787828167.075170:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828167.075171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071232:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828167.075178:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828167.075189:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828167.075201:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333072000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828167.075223:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828167.075307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075320:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158df80 x1874663333071360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075330:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333071360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075351:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075354:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828167.075359:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828167.075377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075385:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158ca80 x1874663333071488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075393:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333071488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075401:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075404:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828167.075408:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828167.075420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075426:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158f100 x1874663333071616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075451:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333071616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075461:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075464:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333071616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828167.075469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333071744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828167.075487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075496:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158e300 x1874663333071744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075514:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333071744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075521:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075524:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075526:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333071744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828167.075531:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333071872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828167.075547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075555:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158dc00 x1874663333071872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075565:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333071872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075571:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075574:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075577:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333071872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828167.075582:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333072000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828167.075619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828167.075654:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828167] req@ffff9057f158c000 x1874663333072000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828167.075663:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333072000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828167.075673:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828167.075675:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828167.075677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333072000:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828167.458443:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828168.482479:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828169.506464:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828171.554615:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828172.194331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194344:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828172.194349:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194363:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194378:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194387:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828172.194438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333072128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194472:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828172.194479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194484:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194487:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194502:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828172.194513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333072256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828172.194530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194534:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194538:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194542:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828172.194553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333072384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828172.194585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194590:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194596:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194605:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828172.194626:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333072512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194635:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828172.194640:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194644:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194646:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194650:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194656:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828172.194667:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333072640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194676:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194679:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828172.194681:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194686:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194689:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194692:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828172.194704:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333072768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828172.194717:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194720:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194722:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194725:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828172.194735:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333072896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828172.194743:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828172.194745:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828172.194748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828172.194751:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9982 00000100:00080000:1.0:1787828172.194754:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828172.194757:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828172.194760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828172.194773:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333073024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828172.578631:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828173.602381:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828174.626504:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828175.650574:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828177.314605:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.314636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828177.314642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.314656:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.314699:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.314724:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828177.314774:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333073152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.314811:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.314817:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828177.314819:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.314825:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.314830:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.314836:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.314849:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828177.314867:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333073280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.314884:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.314910:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828177.314913:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.314934:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.314937:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.314943:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.314949:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828177.314963:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333073408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.315003:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.315008:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828177.315010:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.315015:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.315021:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.315027:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.315034:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828177.315051:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333073536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.315067:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.315072:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828177.315075:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.315083:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.315100:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.315106:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.315113:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828177.315149:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333073664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.315182:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.315187:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828177.315190:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.315197:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.315205:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.315214:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.315313:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828177.315346:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333073792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.315386:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.315391:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828177.315393:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.315400:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.315410:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.315417:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.315433:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828177.315478:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333073920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828177.315503:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828177.315508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828177.315510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828177.315515:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9988 00000100:00080000:0.0:1787828177.315520:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828177.315562:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828177.315571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828177.315598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333074048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828177.698508:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828178.722463:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828179.746674:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828180.386862:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1787828180.386903:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828180.386919:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.386977:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.386985:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.386991:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.386997:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.387003:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.387009:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828180.387014:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00010000:00010000:0.0:1787828181.794483:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828182.434425:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.434440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828182.434446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.434461:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.434477:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.434510:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828182.434587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333074176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.434628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.434635:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828182.434638:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.434646:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.434654:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.434663:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.434682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828182.434715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333074304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.434741:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.434748:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828182.434752:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.434758:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.434769:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.434777:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.434808:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828182.434839:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333074432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.434863:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.434885:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828182.434891:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.434903:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.434912:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.434927:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.434967:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828182.435003:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333074560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.435026:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.435040:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828182.435049:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.435062:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.435073:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.435095:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.435114:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828182.435148:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333074688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.435393:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.435402:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828182.435406:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.435420:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.435439:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.435452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.435463:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828182.435531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333074816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.435580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.435595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828182.435604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.435620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.435634:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.435649:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.435662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828182.435693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333074944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828182.435721:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828182.435733:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828182.435739:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828182.435770:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9993 00000100:00080000:3.0:1787828182.435779:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828182.435789:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828182.435800:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828182.435828:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333075072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828182.818576:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828183.842809:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828184.866395:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828185.890552:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828187.554365:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554374:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828187.554377:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554385:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554394:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554401:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828187.554436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333075200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554478:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554481:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828187.554483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554489:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554499:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828187.554512:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333075328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828187.554524:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554531:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554536:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828187.554553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333075456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828187.554564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554567:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554570:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828187.554582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333075584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554588:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828187.554591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554593:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554595:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554597:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828187.554605:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333075712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828187.554613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554616:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554617:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828187.554636:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333075840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828187.554645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554648:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554651:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554653:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828187.554664:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333075968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828187.554672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828187.554676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828187.554680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828187.554691:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 9998 00000100:00080000:1.0:1787828187.554698:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828187.554706:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828187.554718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828187.554738:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333076096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828187.938394:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828188.962537:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828189.986394:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828192.034395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828192.674330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674337:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828192.674341:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674349:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674358:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674364:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828192.674400:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333076224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674432:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674435:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828192.674437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674440:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674444:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674447:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828192.674466:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333076352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674481:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674483:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828192.674485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674487:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674490:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674492:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674495:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828192.674504:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333076480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674510:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674513:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828192.674514:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674516:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674519:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674525:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828192.674539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333076608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674545:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828192.674549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674552:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674562:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674568:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828192.674581:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333076736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674594:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828192.674604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674608:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674611:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674616:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674622:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828192.674634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333076864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674640:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674643:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828192.674646:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674655:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674658:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674661:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828192.674676:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333076992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828192.674693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828192.674699:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828192.674701:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828192.674707:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10003 00000100:00080000:0.0:1787828192.674712:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828192.674717:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828192.674723:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828192.674734:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333077120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828193.058381:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828194.082486:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828195.106368:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828196.130365:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828197.794310:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794318:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828197.794321:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794330:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794338:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794344:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828197.794378:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333077248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794419:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794422:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828197.794423:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794426:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794430:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794443:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828197.794457:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333077376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794477:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828197.794479:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794485:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794489:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794493:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794499:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828197.794510:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333077504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794517:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828197.794531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794535:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794540:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794544:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828197.794559:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333077632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828197.794579:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794584:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794589:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794595:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828197.794608:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333077760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794619:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794624:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828197.794628:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794633:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794638:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794640:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794643:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828197.794653:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333077888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794671:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794676:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828197.794679:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794688:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794695:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794703:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828197.794726:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333078016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828197.794747:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828197.794750:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828197.794755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828197.794761:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10008 00000100:00080000:0.0:1787828197.794765:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828197.794772:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828197.794780:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828197.794798:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333078144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828198.178359:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828199.202360:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828200.226400:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828202.274467:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828202.914555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333078272:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828202.914777:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.914846:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de0700 x1874663333078272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.914862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333078272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.914870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.914873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.914876:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333078272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828202.914884:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.914927:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.914936:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de2a00 x1874663333078400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.914944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333078400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.914950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.914952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.914954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.914958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.914969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.914975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de1180 x1874663333078528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.914987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333078528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.914993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.914995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.914996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.915017:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828202.915028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.915035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de1880 x1874663333078656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.915042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333078656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.915047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.915049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.915050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828202.915054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828202.915064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.915070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de3800 x1874663333078784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.915077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333078784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.915086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.915088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.915090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333078784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828202.915093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333078912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.915110:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.915116:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de1f80 x1874663333078912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.915123:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333078912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.915128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.915129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.915131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333078912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.915134:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.915144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.915149:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de3100 x1874663333079040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.915156:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333079040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.915160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.915162:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.915165:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.915168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828202.915190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828202.915195:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff9057d0de0e00 x1874663333079168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.915202:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333079168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.915207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.915209:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828202.915210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079168:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828203.298337:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828204.322380:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828205.346383:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828206.370394:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828207.394405:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828208.034446:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333079296:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787828208.034512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034559:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de1500 x1874663333079296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034601:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333079296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034620:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034627:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333079296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828208.034660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828208.034692:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034705:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de1c00 x1874663333079424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034715:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333079424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034723:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828208.034737:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828208.034758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034770:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de0000 x1874663333079552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034779:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333079552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034784:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034789:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828208.034793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828208.034807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de0a80 x1874663333079680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034822:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333079680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828208.034834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828208.034844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de0e00 x1874663333079808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333079808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034863:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333079808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828208.034868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828208.034879:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de3100 x1874663333079936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034890:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333079936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034904:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333079936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828208.034908:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333080064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828208.034918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de1f80 x1874663333080064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333080064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333080064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828208.034951:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333080192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828208.034961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828208.034967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9057d0de3800 x1874663333080192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828208.034974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333080192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828208.034979:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828208.034981:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828208.034983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333080192:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828208.418434:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828209.442352:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828210.466391:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828212.514380:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828213.154330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154338:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828213.154341:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154365:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828213.154409:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333080320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828213.154436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154439:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154452:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154455:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154463:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828213.154472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333080448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154485:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828213.154488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154499:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154501:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154504:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828213.154526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333080576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828213.154536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154541:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828213.154553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333080704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828213.154563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154576:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154578:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828213.154589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333080832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828213.154599:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154601:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154604:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154607:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828213.154619:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333080960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828213.154649:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154653:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154655:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154658:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828213.154669:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333081088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828213.154696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828213.154699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828213.154701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828213.154708:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10023 00000100:00080000:1.0:1787828213.154710:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828213.154714:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828213.154719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828213.154728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333081216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828213.538387:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828214.562395:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828215.586429:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828216.610383:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828217.634365:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828218.274335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274346:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828218.274351:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828218.274419:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333081344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828218.274456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274469:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828218.274495:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333081472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274519:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828218.274522:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274527:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274531:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274536:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274541:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828218.274554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333081600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828218.274574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274579:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274584:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828218.274612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333081728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274640:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828218.274654:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274665:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274670:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274675:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828218.274701:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333081856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274726:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828218.274731:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274741:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274748:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274756:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274765:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828218.274786:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333081984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274803:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274810:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828218.274813:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274824:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274831:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274840:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274852:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828218.274871:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333082112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828218.274884:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828218.274894:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828218.274900:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828218.274909:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10029 00000100:00080000:1.0:1787828218.274917:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828218.274928:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828218.274935:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828218.274953:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333082240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828218.658394:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828219.682350:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828220.706384:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828222.754473:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828223.394320:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394328:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828223.394331:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394339:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394348:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394354:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828223.394385:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333082368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394407:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394409:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828223.394410:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394413:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394416:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394419:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394426:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828223.394435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333082496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394442:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394445:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828223.394446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394455:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828223.394465:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333082624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394473:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394475:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828223.394477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394479:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394482:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394484:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394487:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828223.394494:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333082752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828223.394505:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394507:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394527:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394531:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828223.394553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333082880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828223.394572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394582:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394590:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828223.394621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333083008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394636:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394642:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828223.394646:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394651:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394654:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394657:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828223.394672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333083136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828223.394683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828223.394691:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828223.394695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828223.394703:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10034 00000100:00080000:3.0:1787828223.394708:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828223.394715:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828223.394722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828223.394733:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333083264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828223.778366:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828224.802380:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828225.826347:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828226.850477:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828227.874378:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828228.514271:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514281:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828228.514285:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514296:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514307:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514314:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828228.514353:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333083392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514378:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514381:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828228.514382:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514385:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514389:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514392:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514401:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828228.514412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333083520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514424:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514427:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828228.514428:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514431:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514436:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514439:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828228.514449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333083648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514458:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828228.514459:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514462:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514480:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514483:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514487:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828228.514497:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333083776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828228.514514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514517:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514519:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514521:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828228.514536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333083904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828228.514566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514569:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514571:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514573:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828228.514587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333084032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828228.514605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514629:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828228.514642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333084160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828228.514655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828228.514658:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828228.514660:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828228.514663:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10039 00000100:00080000:3.0:1787828228.514668:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828228.514679:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828228.514683:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828228.514695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333084288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828228.898452:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828229.922467:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828230.946413:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828232.994431:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828233.634398:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634409:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828233.634414:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634424:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634455:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634464:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828233.634533:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333084416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634566:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634570:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828233.634571:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634575:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634579:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634583:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634594:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.634622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333084544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634636:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634650:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.634653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.634679:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333084672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634685:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634689:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828233.634691:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634694:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634696:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634699:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634704:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828233.634714:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333084800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634724:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634727:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.634729:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634734:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634737:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634741:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634745:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828233.634754:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333084928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634762:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634764:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828233.634766:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634776:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634783:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634795:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634803:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.634814:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333085056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634879:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634882:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.634886:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634899:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634907:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634915:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634919:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.634928:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333085184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.634937:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.634940:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.634942:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.634946:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10044 00000100:00080000:0.0:1787828233.634952:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.634961:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828233.634965:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828233.634978:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333085312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828234.018385:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828235.042510:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828236.066447:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828237.090486:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828238.114547:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828238.754312:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754335:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828238.754339:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754362:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754370:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828238.754415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333085440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754448:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828238.754455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754459:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754462:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754466:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754477:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.754492:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333085568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754515:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.754518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754532:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.754560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333085696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828238.754577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754581:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754585:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754588:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754594:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828238.754611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333085824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754635:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.754642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754648:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754654:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754664:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754675:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828238.754698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333085952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754712:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754723:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828238.754727:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754739:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754747:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754753:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754759:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.754780:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333086080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754809:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.754815:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754819:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754822:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754826:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754831:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.754846:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333086208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.754887:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.754901:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.754908:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.754919:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10049 00000100:00080000:0.0:1787828238.754932:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.754945:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828238.754954:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828238.754984:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333086336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828239.138566:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828240.162413:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828241.186441:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828243.234387:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828243.874315:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874323:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828243.874327:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874336:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874345:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874351:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828243.874384:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333086464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874404:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874407:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828243.874408:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874411:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874414:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874418:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874425:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.874436:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333086592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874452:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828243.874453:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874455:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874460:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874462:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.874471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333086720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874477:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874480:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828243.874481:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874484:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874487:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874489:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874492:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828243.874500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333086848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828243.874510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874512:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874514:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874516:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874524:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828243.874533:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333086976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874540:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874543:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828243.874547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874551:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874555:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874558:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.874571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333087104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874598:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874601:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828243.874604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874612:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874616:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874620:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874624:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.874636:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333087232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.874646:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.874649:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828243.874651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.874656:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10054 00000100:00080000:0.0:1787828243.874663:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.874666:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828243.874670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828243.874680:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333087360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828243.874713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333087360:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828244.258457:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828245.282568:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828246.306556:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828246.946562:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828246.946637:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828246.946662:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946725:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946743:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946760:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946777:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946799:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828246.946815:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158c700 x1874663333086976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828246.946816:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828246.946856:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828246.946880:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333086976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.946901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.946909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.946916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333086976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828246.946929:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158ce00 x1874663333087104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.946950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333087104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.946963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.946977:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.946982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333087104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828246.946993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158ea00 x1874663333086592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333086592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333086592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828246.947052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158fb80 x1874663333086720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947071:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333086720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333086720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828246.947099:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158ed80 x1874663333086848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947114:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333086848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947135:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333086848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828246.947144:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158d500 x1874663333086464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947162:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333086464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947178:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947182:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333086464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828246.947254:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158d180 x1874663333087232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947272:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333087232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947287:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947291:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333087232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828246.947300:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828246] req@ffff9057f158d880 x1874663333087360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828246.947314:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333087360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828246.947323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828246.947328:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828246.947331:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333087360:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828247.330400:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828248.354404:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828248.994324:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994332:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828248.994335:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994343:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994352:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994358:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828248.994390:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333087488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.994417:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994419:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828248.994421:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994424:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994426:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994428:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994436:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828248.994445:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333087616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.994453:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828248.994456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828248.994463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333087488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828248.994468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994475:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828248.994481:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333087744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.994487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828248.994491:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994493:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994496:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994499:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994503:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828248.994510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333087872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.994515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828248.994518:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994521:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994524:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994526:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828248.994538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333088000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828248.994538:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828248.994547:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828248.994551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994555:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994558:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994561:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828248.994589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333088128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828248.994590:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78aaa00 x1874663333087488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.994628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828248.994635:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828248.994638:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333087488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.994642:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00080000:1.0:1787828248.994649:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994658:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828248.994660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828248.994664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828248.994668:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.994676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333087488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828248.994682:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333088256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.994694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828248.994698:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828248.994700:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828248.994707:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10059 00000100:00100000:2.0:1787828248.994708:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087616:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828248.994712:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828248.994719:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828248.994724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828248.994738:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333088384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828248.994745:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.994768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78a8a80 x1874663333087616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.994784:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333087616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.994793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.994796:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.994800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828248.994806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828248.994834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.994847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78ab100 x1874663333087744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.994858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333087744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.994866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.994869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.994873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828248.994880:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828248.994899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.994910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78a9500 x1874663333087872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.994921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333087872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.994929:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.994932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.994935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333087872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828248.994941:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828248.994958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.994970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78a8700 x1874663333088000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.994981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333088000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.994989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.994992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.994995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828248.995000:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828248.995019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.995048:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78a9c00 x1874663333088128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.995069:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333088128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.995078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.995081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.995085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828248.995092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828248.995113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.995125:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78aa300 x1874663333088256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.995137:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333088256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.995145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.995148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.995152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828248.995173:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828248.995195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828248.995209:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff9057c78a9180 x1874663333088384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828248.995257:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333088384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828248.995273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828248.995277:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828248.995280:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333088384:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828249.378398:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828250.402477:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828251.426442:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828252.450442:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828253.474440:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828254.114408:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114420:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828254.114425:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114437:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114450:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114460:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828254.114507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333088512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828254.114560:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114565:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114570:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114574:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.114586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828254.114601:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333088640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.114620:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.114635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828254.114650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333088768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828254.114679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114685:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114693:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828254.114695:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333088512:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828254.114703:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.114714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828254.114731:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333088896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114745:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114749:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.114752:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114757:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114779:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114784:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.114790:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828254.114793:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828254.114811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333089024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114828:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114832:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828254.114834:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114841:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.114847:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114853:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.114860:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828254.114871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb222a00 x1874663333088512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828254.114878:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333089152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.114940:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.114946:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.114950:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.114956:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00100000:2.0:1787828254.114963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333088512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828254.114964:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.114978:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828254.115013:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828254.115034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828254.115044:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333089280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828254.115051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115067:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333088512:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828254.115077:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828254.115086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088640:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828254.115090:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.115098:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.115110:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10064 00000100:00080000:3.0:1787828254.115128:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828254.115136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828254.115147:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb222d80 x1874663333088640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828254.115163:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828254.115190:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333088640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828254.115200:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333089408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828254.115217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115295:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115304:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.115315:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828254.115344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115360:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb221c00 x1874663333088768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115374:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333088768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115389:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115393:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.115427:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828254.115449:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115462:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb222300 x1874663333088896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115476:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333088896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115485:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115489:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115492:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333088896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828254.115499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828254.115518:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115530:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb220380 x1874663333089024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115544:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333089024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115553:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115557:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115560:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828254.115566:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828254.115585:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115596:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb220700 x1874663333089152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115610:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333089152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115622:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115626:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.115632:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828254.115650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115661:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb221880 x1874663333089280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115673:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333089280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115691:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.115698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828254.115716:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828254.115726:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9057fb223100 x1874663333089408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.115739:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333089408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.115748:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.115752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828254.115756:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333089408:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828254.498333:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828255.522561:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828256.546445:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828257.570382:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828258.594416:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828259.234305:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234318:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828259.234323:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234336:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234351:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234361:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828259.234402:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333089536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234428:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234431:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828259.234433:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234436:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234439:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234442:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234451:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828259.234460:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333089664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234469:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828259.234470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234481:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828259.234488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333089792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828259.234499:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234502:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234504:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234507:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828259.234518:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333089920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234525:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828259.234528:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234538:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828259.234544:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333089536:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828259.234545:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234556:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828259.234566:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333090048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234577:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828259.234580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234584:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234588:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234591:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828259.234604:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333090176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828259.234616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10069 00000800:00000100:2.0:1787828259.234629:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828259.234631:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234636:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828259.234651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333090304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828259.234659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828259.234662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828259.234664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828259.234670:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10069 00000100:00080000:3.0:1787828259.234675:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828259.234679:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828259.234682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828259.234688:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb223100 x1874663333089536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828259.234693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333090432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828259.234746:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333089536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.234757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.234763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.234767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333089536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828259.234773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828259.234789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.234803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb221880 x1874663333089664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.234811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333089664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.234816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.234819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.234821:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828259.234825:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828259.234835:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.234842:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb220700 x1874663333089792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.234849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333089792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.234854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.234856:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.234858:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828259.234877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828259.234889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.234895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb220380 x1874663333089920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.234903:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333089920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.234907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.234909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.234911:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333089920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828259.234915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828259.234924:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.234930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb222300 x1874663333090048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.234937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333090048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.234943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.234944:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.234946:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828259.234950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828259.234976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.234983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb221c00 x1874663333090176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.234990:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333090176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.235000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.235002:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.235004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828259.235008:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828259.235019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.235025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb222d80 x1874663333090304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.235032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333090304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.235037:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.235039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.235040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828259.235044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828259.235054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828259.235059:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828259/real 1787828259] req@ffff9057fb222a00 x1874663333090432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828259.235066:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333090432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828259.235070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828259.235072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828259.235075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333090432:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828259.618471:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828260.642437:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828261.666417:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828262.690479:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828263.714553:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828264.354340:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354357:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828264.354365:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354381:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.354397:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.354410:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828264.354469:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333090560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354519:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828264.354531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354540:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.354546:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.354552:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.354564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828264.354584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333090688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354599:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828264.354608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354612:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.354617:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.354623:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.354629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828264.354660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333090560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828264.354662:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333090816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354690:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828264.354697:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.354707:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.354711:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.354718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828264.354742:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333090944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354761:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828264.354772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354780:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.354788:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828264.354791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828264.354802:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.354815:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828264.354839:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333091072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354853:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.354859:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828264.354862:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354871:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00100000:2.0:1787828264.354873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb223800 x1874663333090560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828264.354877:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.354885:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.354892:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828264.354913:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333091200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828264.354953:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828264.354963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333090560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828264.354963:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828264.354971:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.354987:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.355000:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828264.355007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828264.355009:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828264.355026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828264.355027:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828264.355039:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333090560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828264.355055:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333091328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828264.355057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090688:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828264.355088:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828264.355093:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828264.355096:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828264.355102:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10075 00000100:00080000:3.0:1787828264.355108:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828264.355113:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828264.355122:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828264.355137:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828264.355151:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333091456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828264.355172:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb222680 x1874663333090688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333090688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355271:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355276:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828264.355309:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828264.355343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355359:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb220e00 x1874663333090816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355374:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333090816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355388:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355392:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828264.355398:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828264.355418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355429:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb220a80 x1874663333090944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355443:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333090944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355456:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355460:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333090944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828264.355466:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828264.355483:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355494:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb221180 x1874663333091072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355516:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333091072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355529:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355533:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828264.355539:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828264.355558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355569:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb220000 x1874663333091200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355589:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333091200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355604:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355616:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828264.355628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828264.355665:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355688:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb223b80 x1874663333091328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355705:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333091328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355723:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828264.355739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828264.355765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828264.355778:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828264/real 1787828264] req@ffff9057fb221500 x1874663333091456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828264.355801:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333091456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828264.355816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828264.355823:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828264.355830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333091456:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828264.738358:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828265.762605:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828266.786373:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828267.810458:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828268.834450:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828269.474441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828269.474459:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474473:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474486:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474496:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828269.474550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333091584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828269.474608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474614:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474618:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474623:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474634:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828269.474650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333091712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474669:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828269.474671:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474676:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474679:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474684:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474690:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828269.474704:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333091840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474716:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474720:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828269.474722:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474727:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474730:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474734:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474740:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828269.474756:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333091584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828269.474759:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333091968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474795:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474799:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828269.474801:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474805:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474810:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474815:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474821:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828269.474836:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333092096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474847:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474852:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828269.474854:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.474860:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474868:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474874:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474880:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828269.474919:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333092224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.474931:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.474936:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828269.474945:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828269.474950:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828269.474959:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.474968:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.474974:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.474982:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828269.475007:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333092352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828269.475026:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828269.475031:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828269.475033:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828269.475039:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10080 00000100:00080000:0.0:1787828269.475044:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828269.475051:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828269.475062:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828269.475091:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333092480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828269.475093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158c000 x1874663333091584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.475195:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333091584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.475216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.475240:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.475246:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333091584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828269.475261:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828269.475396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.475435:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158dc00 x1874663333091712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.475469:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333091712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.475481:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.475486:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.475490:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828269.475499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828269.475553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.475573:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158e300 x1874663333091840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.475592:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333091840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.475604:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.475610:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.475615:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828269.475664:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828269.475701:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.475720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158f100 x1874663333091968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.475741:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333091968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.475755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.475761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.475767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333091968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828269.475779:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828269.475827:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.475843:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158ca80 x1874663333092096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.475859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333092096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.475871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.475878:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.475927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828269.475939:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828269.475972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.475992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158df80 x1874663333092224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.476010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333092224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.476046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.476051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.476055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828269.476064:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828269.476092:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.476109:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158e680 x1874663333092352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.476149:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333092352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.476163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.476169:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.476174:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828269.476186:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828269.476219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828269.476232:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828269/real 1787828269] req@ffff9057f158f800 x1874663333092480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828269.476294:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333092480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828269.476307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828269.476311:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828269.476318:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333092480:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828269.858560:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828270.882410:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828271.906524:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828272.930635:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828273.954550:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828274.594380:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594394:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828274.594400:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.594415:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.594431:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828274.594441:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828274.594499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333092608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.594535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828274.594541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.594546:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.594550:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828274.594554:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.594567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828274.594585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333092736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.594602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828274.594610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.594616:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00100000:2.0:1787828274.594622:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333092608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828274.594625:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828274.594642:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.594652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828274.594675:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333092864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.594688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828274.594698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.594704:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.594713:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828274.594719:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.594729:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828274.594730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828274.594757:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333092992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.594773:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594777:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828274.594779:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.594785:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.594791:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828274.594797:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78aa680 x1874663333092608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.594798:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.594809:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828274.594835:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333093120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828274.594835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333092608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.594905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.594908:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.594915:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828274.594920:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.594920:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.594933:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333092608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828274.594934:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.594946:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828274.594954:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828274.594956:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.594971:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828274.594988:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333093248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.594998:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828274.595004:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828274.595007:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828274.595012:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.595019:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.595024:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828274.595029:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828274.595034:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828274.595039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78a9180 x1874663333092736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828274.595050:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333093376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.595081:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828274.595084:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828274.595089:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333092736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.595090:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828274.595103:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10085 00000100:00080000:1.0:1787828274.595111:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828274.595114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.595118:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828274.595124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787828274.595126:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828274.595133:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828274.595145:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333093504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828274.595174:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828274.595206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595255:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78aa300 x1874663333092864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595277:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333092864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595310:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828274.595324:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828274.595350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595362:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78a9c00 x1874663333092992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595374:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333092992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595387:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595390:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333092992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828274.595396:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828274.595413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595421:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78a8700 x1874663333093120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595440:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333093120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595450:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595453:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828274.595458:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828274.595473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595494:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78a9500 x1874663333093248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595504:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333093248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595514:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595518:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828274.595523:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828274.595549:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595565:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78ab100 x1874663333093376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595584:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333093376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595607:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595611:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595614:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828274.595621:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828274.595644:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828274.595655:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff9057c78a8a80 x1874663333093504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.595667:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333093504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.595675:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828274.595680:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828274.595685:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333093504:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828274.978459:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828276.002449:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828277.026438:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828278.050392:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828279.074440:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828279.714311:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714329:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828279.714333:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714345:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714357:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714365:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828279.714408:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333093632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828279.714434:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828279.714440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714445:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714465:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828279.714480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333093760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828279.714482:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333093632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828279.714491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828279.714497:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828279.714521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333093888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828279.714529:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828279.714534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714537:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714539:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714548:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828279.714556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333094016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828279.714558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828279.714565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714568:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828279.714569:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714575:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714579:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714583:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828279.714593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333094144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828279.714602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714605:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828279.714607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714612:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828279.714617:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78aaa00 x1874663333093632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.714620:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828279.714638:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333094272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828279.714638:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333093632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.714648:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828279.714651:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.714652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828279.714653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828279.714655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828279.714657:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00100000:2.0:1787828279.714660:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333093632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828279.714661:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714665:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828279.714677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333094400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828279.714687:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093760:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828279.714694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828279.714697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828279.714699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828279.714702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10090 00000100:00080000:1.0:1787828279.714706:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828279.714709:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828279.714709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787828279.714714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828279.714723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78a9f80 x1874663333093760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828279.714724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333094528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828279.714737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333093760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.714745:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.714748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.714751:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828279.714757:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828279.714778:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.714788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78a8380 x1874663333093888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.714796:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333093888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.714802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.714805:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.714809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333093888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828279.714830:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828279.714852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.714865:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78a9880 x1874663333094016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.714878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333094016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.714887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.714890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.714894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828279.714900:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828279.714922:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.714931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78ab800 x1874663333094144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.714939:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333094144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.714952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.714954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.714957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828279.714961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828279.714974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.714980:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78abb80 x1874663333094272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.714988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333094272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.714998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.715000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.715002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828279.715015:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828279.715025:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.715037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78a8000 x1874663333094400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.715045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333094400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.715050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.715053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.715055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828279.715058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828279.715068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828279.715074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff9057c78aad80 x1874663333094528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828279.715081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333094528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828279.715086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828279.715088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828279.715091:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333094528:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828280.098679:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828281.122446:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828282.146470:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828283.170417:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828284.194360:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828284.834374:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834384:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828284.834388:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834399:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834410:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834418:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828284.834461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333094656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828284.834493:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834499:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834503:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828284.834523:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333094784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834534:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828284.834535:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834538:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834540:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834543:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828284.834553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333094912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828284.834564:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333094656:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828284.834568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834581:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834586:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828284.834606:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333095040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828284.834623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834627:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834630:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834634:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828284.834647:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828284.834651:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333095168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828284.834663:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834667:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834670:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828284.834683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333095296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828284.834690:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828284.834695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828284.834698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78ab480 x1874663333094656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828284.834701:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834706:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834712:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828284.834719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828284.834727:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333094656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828284.834730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333095424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828284.834755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828284.834756:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828284.834761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828284.834763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828284.834765:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10095 00000100:00080000:1.0:1787828284.834767:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828284.834773:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828284.834774:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787828284.834792:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828284.834798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333094656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828284.834804:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333095552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828284.834813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828284.834838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.834848:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78aad80 x1874663333094784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.834857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333094784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.834864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.834866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.834868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828284.834890:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828284.834904:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.834911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78a8000 x1874663333094912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.834919:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333094912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.834924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.834926:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.834928:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333094912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828284.834931:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828284.834942:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.834947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78abb80 x1874663333095040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.834955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333095040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.834960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.834961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.834963:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828284.834966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828284.834976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.834981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78ab800 x1874663333095168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.834994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333095168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.834999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.835001:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.835003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828284.835006:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828284.835016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.835021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78a9880 x1874663333095296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.835027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333095296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.835032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.835034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.835035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828284.835039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828284.835048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.835054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78a8380 x1874663333095424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.835060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333095424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.835069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.835072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.835074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828284.835077:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828284.835087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828284.835093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828284] req@ffff9057c78a9f80 x1874663333095552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828284.835100:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333095552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828284.835105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828284.835107:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828284.835109:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333095552:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828285.218399:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828286.242360:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828287.266577:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828288.290368:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828289.314418:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828289.954356:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954366:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828289.954371:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954383:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954395:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.954403:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828289.954447:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333095680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.954469:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954472:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828289.954474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954477:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954479:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.954482:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954491:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.954502:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333095808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828289.954509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333095680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828289.954513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.954518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954521:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954523:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.954526:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954529:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.954544:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333095936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.954550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828289.954555:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954558:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954561:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.954563:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954567:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828289.954575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333096064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.954583:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828289.954583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828289.954588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.954590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954594:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954597:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.954600:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954604:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828289.954611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333096192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.954617:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954620:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828289.954622:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954624:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954627:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828289.954631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158f800 x1874663333095680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828289.954632:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954639:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.954651:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333096320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828289.954657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333095680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828289.954662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954670:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.954674:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954679:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:2.0:1787828289.954679:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828289.954683:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828289.954687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828289.954690:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954695:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333095680:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828289.954696:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.954707:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333096448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.954715:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.954717:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.954718:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.954721:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10100 00000100:00080000:0.0:1787828289.954723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828289.954726:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095808:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828289.954729:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828289.954735:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828289.954758:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333096576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828289.954765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158e680 x1874663333095808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954790:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333095808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.954796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.954799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.954802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828289.954806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828289.954817:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954823:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158df80 x1874663333095936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954831:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333095936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.954836:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.954844:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.954847:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333095936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828289.954850:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828289.954864:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954870:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158ca80 x1874663333096064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333096064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.954884:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.954885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.954887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828289.954891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828289.954901:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158f100 x1874663333096192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333096192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.954920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.954927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.954929:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828289.954933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828289.954944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158e300 x1874663333096320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333096320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.954968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.954970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.954973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828289.954976:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828289.954986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.954991:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158dc00 x1874663333096448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.954999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333096448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.955009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.955011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.955014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828289.955029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828289.955041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828289.955048:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff9057f158c000 x1874663333096576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828289.955055:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333096576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828289.955061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828289.955063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828289.955065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333096576:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828290.338388:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828291.362378:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828292.386346:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828293.410346:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828294.434432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828295.074285:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074294:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828295.074297:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074306:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074314:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074321:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828295.074357:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333096704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828295.074388:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074392:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828295.074394:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074398:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074401:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074404:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074412:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828295.074424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333096832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828295.074433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828295.074436:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333096704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828295.074437:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828295.074441:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074450:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074452:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828295.074464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333096960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828295.074470:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074473:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828295.074474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074477:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074479:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074481:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828295.074491:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333097088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828295.074498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828295.074501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074506:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10105 00000800:00000100:2.0:1787828295.074508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828295.074510:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074514:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074518:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828295.074526:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333097216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828295.074532:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074535:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828295.074537:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074540:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074543:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074547:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074547:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158d880 x1874663333096704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828295.074554:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828295.074569:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333096704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828295.074575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333097344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828295.074581:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828295.074584:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.074588:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828295.074589:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828295.074593:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828295.074594:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333096704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828295.074599:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074603:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074606:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074609:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828295.074616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333097472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828295.074620:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096832:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828295.074633:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828295.074636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828295.074637:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828295.074641:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10105 00000100:00080000:0.0:1787828295.074645:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828295.074647:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828295.074651:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828295.074656:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828295.074663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333097600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828295.074683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158d180 x1874663333096832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074696:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333096832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074719:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074723:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828295.074728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828295.074747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074755:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158d500 x1874663333096960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074763:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333096960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074768:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074771:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333096960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828295.074776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828295.074787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074792:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158ed80 x1874663333097088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074800:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333097088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828295.074811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828295.074821:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074827:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158fb80 x1874663333097216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333097216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828295.074851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828295.074861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074867:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158ea00 x1874663333097344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333097344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074889:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828295.074912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828295.074934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.074945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158ce00 x1874663333097472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.074957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333097472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.074973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.074976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.074978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828295.074983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828295.074998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828295.075006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9057f158c700 x1874663333097600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.075013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333097600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.075019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.075020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828295.075022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333097600:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828295.458370:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828296.482419:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828297.506354:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828298.530371:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828299.554386:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828300.194291:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194298:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828300.194302:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194310:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194326:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194332:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828300.194368:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333097728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.194398:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194401:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.194402:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194405:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194407:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194411:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828300.194421:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.194438:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333097856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828300.194440:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333097728:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828300.194457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194462:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.194463:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194467:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194470:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194474:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828300.194479:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.194490:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333097984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.194497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.194501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194503:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10110 00000800:00000100:2.0:1787828300.194508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828300.194509:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194515:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828300.194519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828300.194527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333098112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.194534:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194536:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.194537:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194548:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194549:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158c700 x1874663333097728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828300.194551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828300.194563:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333098240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828300.194569:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333097728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828300.194574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.194582:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.194585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828300.194589:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828300.194590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194595:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828300.194598:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828300.194602:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194606:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333097728:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828300.194607:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.194620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333098368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.194632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828300.194632:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097856:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828300.194639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.194642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:0.0:1787828300.194649:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.194652:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828300.194656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828300.194657:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828300.194668:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333098496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828300.194683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158ce00 x1874663333097856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828300.194709:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828300.194713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333097856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828300.194714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.194717:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.194724:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10110 00000100:00080000:2.0:1787828300.194728:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828300.194729:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828300.194734:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828300.194736:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097856:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828300.194742:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828300.194747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828300.194755:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333098624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828300.194766:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194775:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158ea00 x1874663333097984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194785:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333097984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194796:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333097984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828300.194801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828300.194812:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158fb80 x1874663333098112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194833:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333098112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194842:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828300.194846:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828300.194856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194862:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158ed80 x1874663333098240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333098240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194878:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828300.194889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828300.194899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158d500 x1874663333098368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333098368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828300.194929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828300.194939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158d180 x1874663333098496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333098496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828300.194963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828300.194973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828300.194978:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828300] req@ffff9057f158d880 x1874663333098624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828300.194985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333098624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828300.194990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828300.194992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828300.194994:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333098624:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828300.578380:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828301.602404:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828302.626446:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828303.650334:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828304.674379:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828305.314291:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314298:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828305.314302:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314310:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314319:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314324:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828305.314356:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333098752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828305.314375:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314378:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828305.314379:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314382:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314384:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314386:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314393:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828305.314402:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333098880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828305.314409:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314412:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828305.314413:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314416:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314418:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314419:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314423:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828305.314426:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333098752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828305.314431:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333099008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828305.314444:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314447:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828305.314448:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314450:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314452:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314455:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314458:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828305.314464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333099136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828305.314471:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314473:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828305.314474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314476:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314479:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314481:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828305.314490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828305.314494:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333099264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828305.314501:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314503:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828305.314505:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314508:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314511:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314514:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828305.314525:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333099392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828305.314529:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158c000 x1874663333098752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828305.314540:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314546:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828305.314547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314551:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314555:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828305.314555:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333098752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828305.314561:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314568:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828305.314573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828305.314577:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333099520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828305.314580:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314589:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333098752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828305.314592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828305.314597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828305.314598:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098880:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828305.314600:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828305.314606:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10116 00000100:00080000:0.0:1787828305.314609:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828305.314612:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828305.314615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828305.314623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828305.314626:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333099648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828305.314649:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158dc00 x1874663333098880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314662:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333098880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314668:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314674:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333098880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828305.314692:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828305.314708:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314715:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158e300 x1874663333099008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314724:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333099008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314729:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314731:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314733:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828305.314737:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828305.314747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158f100 x1874663333099136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314760:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333099136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314765:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314768:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828305.314772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828305.314781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158ca80 x1874663333099264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333099264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828305.314810:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828305.314828:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314835:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158df80 x1874663333099392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314842:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333099392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314847:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314848:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314850:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828305.314854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828305.314864:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158e680 x1874663333099520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314876:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333099520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314885:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828305.314892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828305.314902:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828305.314907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828305/real 1787828305] req@ffff9057f158f800 x1874663333099648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828305.314914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333099648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828305.314918:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828305.314920:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828305.314922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333099648:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828305.698440:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828306.722418:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828307.746348:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828308.770513:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828309.794411:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828310.434325:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434336:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828310.434341:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434378:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434386:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828310.434452:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333099776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434481:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434484:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828310.434486:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434490:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434494:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434497:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828310.434528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333099904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434538:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434542:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828310.434544:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434547:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434551:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434555:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434560:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828310.434568:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333100032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828310.434588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333099776:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828310.434593:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434617:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434621:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434624:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828310.434662:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333100160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434670:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434674:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828310.434675:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434679:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434683:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434686:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434690:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828310.434699:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333100288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434718:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434722:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828310.434723:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434727:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434729:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434733:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434736:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828310.434745:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333100416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434752:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434756:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828310.434758:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434762:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434766:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434770:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828310.434778:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828310.434799:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333100544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828310.434810:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828310.434813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828310.434814:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828310.434818:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10121 00000100:00080000:0.0:1787828310.434822:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828310.434826:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828310.434830:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828310.434841:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333100672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828310.434858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828310.434886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828310.434903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828310.434924:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828310.434944:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828310.434958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100672:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828310.818548:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828311.842429:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828312.866417:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828313.506472:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828313.506510:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828313.506525:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506572:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506599:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506610:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506617:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828313.506625:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158df80 x1874663333100032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828313.506625:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506637:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828313.506648:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828313.506663:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333100032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506675:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506679:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828313.506692:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158ca80 x1874663333100160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506706:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333100160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506715:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506719:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828313.506730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158f100 x1874663333100288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506742:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333100288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828313.506771:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158e300 x1874663333100416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333100416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506795:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828313.506805:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158dc00 x1874663333100544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333100544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506825:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506831:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828313.506837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158c000 x1874663333100672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333100672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506856:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333100672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828313.506869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158f800 x1874663333099776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333099776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333099776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828313.506903:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828310/real 1787828313] req@ffff9057f158e680 x1874663333099904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828313.506913:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333099904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828313.506920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828313.506923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828313.506926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333099904:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1787828313.890355:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828314.914394:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828315.554311:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554322:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828315.554326:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554337:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554348:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554356:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828315.554391:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333100800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828315.554413:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554416:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828315.554428:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554431:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554434:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554438:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554446:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828315.554459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333100928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828315.554461:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333100800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828315.554476:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554479:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828315.554480:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554482:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554485:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554487:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828315.554498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333101056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828315.554504:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828315.554508:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554511:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554514:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554516:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828315.554527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333101184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828315.554533:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554536:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787828315.554536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828315.554540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554544:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554547:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554550:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828315.554560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333101312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828315.554587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554589:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828315.554590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554593:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554596:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554598:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828315.554609:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158d880 x1874663333100800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828315.554612:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333101440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828315.554641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554644:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828315.554646:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828315.554650:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554653:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828315.554656:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333100800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828315.554660:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828315.554674:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.554680:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828315.554685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333101568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828315.554689:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333100800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828315.554711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828315.554715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828315.554717:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828315.554722:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100928:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828315.554723:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10126 00000100:00080000:0.0:1787828315.554730:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828315.554735:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828315.554741:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828315.554756:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333101696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828315.554784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.554807:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158d180 x1874663333100928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.554819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333100928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.554827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.554830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.554833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333100928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828315.554839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828315.554858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.554870:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158d500 x1874663333101056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.554883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333101056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.554896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.554899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.554902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828315.554908:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828315.554927:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.554937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158ed80 x1874663333101184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.554947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333101184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.554954:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.554957:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.554960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828315.554965:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828315.554982:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.554992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158fb80 x1874663333101312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.555003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333101312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.555010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.555013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.555016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828315.555021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828315.555039:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.555051:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158ea00 x1874663333101440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.555061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333101440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.555069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.555072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.555075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828315.555081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828315.555099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.555110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158ce00 x1874663333101568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.555122:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333101568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.555131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.555134:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.555138:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828315.555168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828315.555190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828315.555202:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828315/real 1787828315] req@ffff9057f158c700 x1874663333101696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828315.555215:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333101696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828315.555251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828315.555255:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828315.555258:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333101696:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828315.938525:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828316.962445:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828317.986497:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828319.010391:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828320.034429:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828320.674399:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.674412:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828320.674417:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.674429:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674443:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.674452:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828320.674503:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333101824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828320.674578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.674583:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828320.674585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.674591:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.674600:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828320.674611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828320.674628:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333101952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828320.674661:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.674666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828320.674669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.674674:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674678:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.674683:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828320.674691:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828320.674696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333101824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828320.674708:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333102080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828320.674724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.674730:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828320.674733:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.674741:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674776:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.674785:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828320.674795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828320.674804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828320.674834:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333102208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828320.674849:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.674854:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828320.674857:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.674880:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674885:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828320.674888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb221500 x1874663333101824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828320.674894:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828320.674906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828320.674926:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333102336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828320.674931:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333101824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828320.674959:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828320.674966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828320.674967:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828320.674974:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828320.674979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828320.674985:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.674990:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.674994:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828320.674999:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828320.675004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333101824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828320.675015:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333102464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828320.675021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101952:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828320.675042:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828320.675047:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828320.675050:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.675057:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.675063:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.675071:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828320.675072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828320.675078:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828320.675098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb223b80 x1874663333101952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828320.675099:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333102592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828320.675152:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828320.675159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333101952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828320.675160:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828320.675166:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828320.675177:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10131 00000100:00080000:3.0:1787828320.675185:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828320.675195:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828320.675206:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828320.675212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828320.675223:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333102720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828320.675279:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675287:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333101952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828320.675318:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828320.675366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675383:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb220000 x1874663333102080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675398:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333102080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675413:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675416:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828320.675423:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828320.675444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675455:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb221180 x1874663333102208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675468:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333102208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675478:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675481:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675485:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828320.675492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828320.675511:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675521:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb220a80 x1874663333102336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675579:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333102336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675592:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675595:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828320.675601:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828320.675624:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675655:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb220e00 x1874663333102464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675670:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333102464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675683:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675687:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828320.675694:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828320.675715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675727:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb222680 x1874663333102592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675740:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333102592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675752:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828320.675776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828320.675812:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828320.675834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff9057fb223800 x1874663333102720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828320.675859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333102720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828320.675890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828320.675894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828320.675898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333102720:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828321.058452:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828322.082411:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828323.106592:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828324.130595:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828325.154355:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828325.794338:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794348:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828325.794354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794383:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828325.794433:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333102848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828325.794465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794469:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828325.794471:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794481:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794486:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828325.794510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828325.794522:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333102848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828325.794529:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333102976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828325.794543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828325.794549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794554:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794560:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794564:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828325.794570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828325.794579:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828325.794585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333103104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828325.794604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828325.794610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828325.794614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78aaa00 x1874663333102848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.794619:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794633:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828325.794636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333102848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828325.794641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828325.794646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.794650:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.794664:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333102848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828325.794670:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333103232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828325.794682:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828325.794686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102976:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828325.794688:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828325.794693:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794700:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794706:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828325.794711:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828325.794714:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828325.794733:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828325.794739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78a8a80 x1874663333102976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.794750:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333103360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828325.794759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333102976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.794767:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794773:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828325.794776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828325.794780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.794782:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794786:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794791:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828325.794795:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.794808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333102976:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787828325.794811:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828325.794815:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828325.794831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333103488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828325.794838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828325.794850:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828325.794855:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828325.794856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78ab100 x1874663333103104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.794857:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794865:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794871:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794889:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828325.794893:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828325.794897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333103104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.794903:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333103616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828325.794910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.794916:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828325.794918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828325.794921:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828325.794925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103104:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828325.794926:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828325.794933:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10136 00000100:00080000:1.0:1787828325.794936:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828325.794939:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828325.794942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103232:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1787828325.794953:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828325.794965:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333103744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828325.794973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828325.794988:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78a9500 x1874663333103232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828325.794999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333103232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828325.795005:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.795007:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.795020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828325.795037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828325.795058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828325.795070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78a8700 x1874663333103360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828325.795080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333103360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828325.795086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.795088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.795090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333103360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828325.795095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828325.795126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828325.795137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78a9c00 x1874663333103488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828325.795150:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333103488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828325.795159:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.795162:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.795165:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828325.795169:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828325.795192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828325.795199:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78aa300 x1874663333103616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828325.795207:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333103616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828325.795212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.795214:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.795216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828325.795264:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828325.795277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828325.795285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff9057c78a9180 x1874663333103744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828325.795292:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333103744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828325.795302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828325.795304:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828325.795317:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333103744:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828326.178355:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828327.202374:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828328.226511:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828329.250386:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828330.274575:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828330.914328:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828330.914345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914357:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914370:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914378:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828330.914422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333103872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.914448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828330.914454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914477:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914488:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828330.914500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828330.914502:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333103872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828330.914520:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333104000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.914534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.914540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914544:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914550:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914562:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828330.914568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828330.914574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828330.914585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333104128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.914597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828330.914602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914606:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914611:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914616:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828330.914621:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78aa680 x1874663333103872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828330.914625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828330.914645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333104256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828330.914652:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333103872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828330.914663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.914680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828330.914684:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828330.914687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914693:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828330.914694:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828330.914704:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828330.914705:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333103872:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828330.914717:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828330.914742:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828330.914743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333104384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828330.914775:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828330.914777:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914785:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828330.914789:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914797:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00100000:2.0:1787828330.914802:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78a9180 x1874663333104000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828330.914805:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914816:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828330.914825:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828330.914844:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333104000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828330.914846:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333104512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.914891:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828330.914894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828330.914896:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.914901:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828330.914903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828330.914910:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10141 00000100:00080000:1.0:1787828330.914918:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.914922:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828330.914926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104000:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787828330.914927:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828330.914944:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333104640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828330.914946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104128:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828330.914968:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.914972:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.914975:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.914982:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10141 00000800:00000100:2.0:1787828330.914989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828330.914991:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.915002:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78aa300 x1874663333104128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828330.915014:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828330.915026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333104128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828330.915043:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333104768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828330.915045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828330.915069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828330.915093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915106:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78a9c00 x1874663333104256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.915118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333104256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.915129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915142:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828330.915159:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828330.915177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915186:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78a8700 x1874663333104384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.915198:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333104384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.915209:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915212:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333104384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828330.915217:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828330.915275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915286:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78a9500 x1874663333104512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.915298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333104512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.915309:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915312:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828330.915318:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828330.915333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915341:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78ab100 x1874663333104640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.915352:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333104640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.915369:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915372:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828330.915377:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828330.915391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828330.915399:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff9057c78a8a80 x1874663333104768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.915409:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333104768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.915417:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.915420:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828330.915422:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333104768:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828331.298527:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828332.322405:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828333.346345:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828334.370472:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828335.394507:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828336.034477:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.034490:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828336.034495:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.034508:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.034521:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.034531:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828336.034580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333104896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828336.034631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.034637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828336.034640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.034645:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.034650:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.034655:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.034667:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828336.034685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333105024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828336.034701:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.034705:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828336.034708:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.034713:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.034720:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828336.034723:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333104896:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787828336.034734:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.034748:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828336.034774:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333105152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828336.034796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.034804:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828336.034809:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.034817:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.034825:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828336.034839:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828336.034839:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.034858:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828336.034883:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333105280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828336.034904:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828336.034907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de1880 x1874663333104896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828336.034917:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828336.034927:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.034944:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00100000:1.0:1787828336.034954:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333104896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828336.034956:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.034976:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828336.034981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828336.034987:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828336.034992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333104896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828336.035018:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333105408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828336.035025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105024:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787828336.035044:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.035051:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828336.035054:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.035061:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.035067:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.035073:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.035080:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828336.035102:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333105536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828336.035105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828336.035125:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.035131:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828336.035134:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828336.035139:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.035144:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.035147:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.035152:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828336.035176:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333105664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828336.035178:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de1180 x1874663333105024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828336.035203:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828336.035214:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828336.035279:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828336.035284:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333105024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828336.035290:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10146 00000100:00080000:2.0:1787828336.035298:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828336.035309:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828336.035323:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828336.035331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828336.035344:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333105792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.035358:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035379:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828336.035449:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828336.035497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035516:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de2a00 x1874663333105152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035532:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333105152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035547:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035552:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828336.035560:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828336.035582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035594:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de0700 x1874663333105280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035608:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333105280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035618:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035622:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035625:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828336.035632:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828336.035653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035664:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de3800 x1874663333105408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035678:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333105408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035688:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333105408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828336.035701:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828336.035720:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035731:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de1f80 x1874663333105536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035745:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333105536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828336.035767:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828336.035787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de3100 x1874663333105664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333105664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828336.035844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828336.035863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828336.035874:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9057d0de0e00 x1874663333105792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.035887:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333105792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.035896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.035900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828336.035903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333105792:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828336.418466:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828337.442611:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828338.466463:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828339.490594:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828340.514449:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828341.154430:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828341.154448:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.154461:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.154475:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.154484:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828341.154533:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333105920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828341.154575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828341.154583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.154588:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.154593:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.154598:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.154611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828341.154630:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333106048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828341.154641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333105920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828341.154651:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154658:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828341.154660:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.154667:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.154672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.154676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.154682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828341.154701:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333106176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828341.154713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828341.154721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.154726:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.154732:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828341.154740:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828341.154742:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.154756:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828341.154775:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333106304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828341.154789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154793:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828341.154796:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.154804:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00100000:2.0:1787828341.154806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78aaa00 x1874663333105920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828341.154809:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.154824:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.154838:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828341.154857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333105920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828341.154861:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333106432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828341.154908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828341.154911:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.154921:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828341.154923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828341.154927:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828341.154937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333105920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828341.154941:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.154955:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.154961:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.154967:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828341.154981:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333106560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828341.154986:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106048:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828341.155016:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828341.155023:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828341.155027:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828341.155036:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10151 00000800:00000100:2.0:1787828341.155042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828341.155046:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828341.155057:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828341.155065:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828341.155071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78a9f80 x1874663333106048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828341.155088:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333106688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828341.155101:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333106048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828341.155127:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828341.155130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155142:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828341.155146:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828341.155152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106048:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828341.155157:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828341.155167:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106176:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828341.155178:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10151 00000100:00080000:1.0:1787828341.155191:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828341.155204:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828341.155206:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155260:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78a8380 x1874663333106176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155279:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333106176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828341.155282:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828341.155303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155309:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828341.155330:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333106816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828341.155330:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828341.155377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155433:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78a9880 x1874663333106304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155449:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333106304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828341.155459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155463:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155467:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828341.155475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828341.155498:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78ab800 x1874663333106432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155524:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333106432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828341.155533:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155537:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155541:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333106432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828341.155561:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828341.155581:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155593:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78abb80 x1874663333106560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155616:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333106560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828341.155626:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155629:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828341.155640:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828341.155659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155670:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78a8000 x1874663333106688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155684:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333106688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828341.155693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155696:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828341.155706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828341.155724:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828341.155734:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9057c78aad80 x1874663333106816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828341.155747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333106816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828341.155764:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828341.155768:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828341.155772:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333106816:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828341.538474:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828342.562483:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828343.586392:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828344.610573:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828345.634568:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828346.274442:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274466:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828346.274473:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828346.274486:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274501:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274510:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828346.274563:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333106944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274607:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274612:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828346.274614:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828346.274620:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274624:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274628:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.274658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828346.274681:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333107072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274697:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828346.274699:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828346.274702:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274705:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274714:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828346.274718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333106944:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828346.274720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828346.274736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333107200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274755:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274759:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828346.274760:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828346.274764:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274768:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274771:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.274776:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828346.274789:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333107328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274798:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274802:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828346.274803:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828346.274812:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828346.274813:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274823:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274827:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.274833:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828346.274846:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333107456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274856:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.274860:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828346.274862:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828346.274869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158c700 x1874663333106944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828346.274873:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274883:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.274894:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.274900:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828346.274911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333106944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828346.274916:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333107584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.274965:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828346.274969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828346.274970:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828346.274977:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828346.274985:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828346.274986:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.274996:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828346.274999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333106944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828346.275005:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.275013:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828346.275018:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828346.275032:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333107712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828346.275043:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828346.275047:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828346.275049:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828346.275053:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10157 00000100:00080000:0.0:1787828346.275060:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828346.275063:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828346.275069:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828346.275076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828346.275101:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333107840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828346.275110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158ce00 x1874663333107072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275155:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333107072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275172:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275177:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828346.275215:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828346.275272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275286:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158ea00 x1874663333107200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275299:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333107200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275311:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275315:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828346.275321:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828346.275338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275373:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158fb80 x1874663333107328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275389:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333107328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275430:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275435:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828346.275443:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828346.275471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275486:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158ed80 x1874663333107456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275501:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333107456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275514:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275517:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333107456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828346.275524:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828346.275544:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275555:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158d500 x1874663333107584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275569:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333107584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275578:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275582:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828346.275592:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828346.275611:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275621:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158d180 x1874663333107712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275634:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333107712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275661:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275666:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275671:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828346.275680:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828346.275709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828346.275727:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828346/real 1787828346] req@ffff9057f158d880 x1874663333107840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828346.275747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333107840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828346.275758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828346.275761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828346.275765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333107840:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828346.658325:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828347.682442:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828348.706369:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828349.730556:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828350.754509:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828351.394338:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828351.394349:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828351.394355:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394365:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:1.0:1787828351.394375:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828351.394421:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333107968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828351.394447:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828351.394451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828351.394453:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394457:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:1.0:1787828351.394460:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394465:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394474:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828351.394490:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333108096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828351.394496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333107968:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828351.394509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828351.394513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828351.394515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:1.0:1787828351.394525:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394529:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394535:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828351.394548:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333108224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828351.394556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828351.394569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828351.394573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828351.394575:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:1.0:1787828351.394584:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394589:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828351.394596:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78ab480 x1874663333107968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828351.394619:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333107968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828351.394621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333108352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828351.394639:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828351.394643:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828351.394644:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.394651:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333107968:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828351.394652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828351.394657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394664:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:1.0:1787828351.394668:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828351.394674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828351.394675:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394685:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828351.394696:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828351.394701:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333108480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828351.394708:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78aad80 x1874663333108096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828351.394720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828351.394725:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333108096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828351.394727:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828351.394732:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828351.394735:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828351.394741:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00080000:2.0:1787828351.394741:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.394748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828351.394750:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828351.394754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108224:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828351.394760:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394767:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828351.394773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.394786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78a8000 x1874663333108224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828351.394787:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333108608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828351.394807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333108224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828351.394812:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828351.394817:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828351.394819:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828351.394822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828351.394824:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828351.394826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108224:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828351.394835:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00100000:2.0:1787828351.394836:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108352:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787828351.394842:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394849:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394855:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828351.394855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.394864:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78abb80 x1874663333108352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828351.394873:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333108736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828351.394877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333108352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828351.394900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828351.394904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828351.394905:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828351.394909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108352:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787828351.394915:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828351.394919:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828351.394924:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10162 00000100:00100000:2.0:1787828351.394927:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108480:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787828351.394930:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828351.394937:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828351.394942:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828351.394943:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.394952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78ab800 x1874663333108480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828351.394960:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333108864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828351.394964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333108480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828351.394988:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828351.394995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.395002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333108480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828351.395009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828351.395022:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.395031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78a9880 x1874663333108608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828351.395038:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333108608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828351.395048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828351.395050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.395052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828351.395055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828351.395065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.395070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78a8380 x1874663333108736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828351.395077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333108736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828351.395081:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828351.395083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.395085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828351.395093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828351.395103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828351.395112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828351/real 1787828351] req@ffff9057c78a9f80 x1874663333108864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828351.395124:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333108864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828351.395132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828351.395135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828351.395139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333108864:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828351.778535:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828352.802448:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828353.826442:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828354.850579:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828355.874528:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828356.514501:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514534:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828356.514541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514558:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514574:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514587:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828356.514644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333108992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828356.514698:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828356.514705:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514726:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514739:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514742:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.514751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828356.514761:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333109120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828356.514768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333108992:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828356.514771:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514776:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828356.514777:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514780:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514782:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514784:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.514788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828356.514797:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333109248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828356.514804:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514806:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828356.514818:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514822:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514831:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514834:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.514838:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828356.514839:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828356.514852:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333109376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828356.514862:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514866:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828356.514868:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514872:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00100000:2.0:1787828356.514877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb222a00 x1874663333108992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828356.514879:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514889:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828356.514896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333108992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828356.514896:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828356.514909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828356.514913:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333109504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828356.514916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.514924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333108992:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828356.514928:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828356.514932:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828356.514933:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514937:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514940:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.514943:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.514947:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828356.514948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828356.514959:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333109632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828356.514969:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828356.514973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828356.514974:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828356.514977:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.514982:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00080000:3.0:1787828356.514986:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828356.514989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb222d80 x1874663333109120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828356.514991:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.514998:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828356.515009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333109120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828356.515030:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333109760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828356.515070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828356.515085:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828356.515094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109120:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828356.515095:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828356.515120:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828356.515136:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10167 00000100:00100000:2.0:1787828356.515147:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109248:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828356.515151:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828356.515164:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828356.515195:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828356.515217:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333109888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828356.515232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515262:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb221c00 x1874663333109248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515274:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333109248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515330:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828356.515337:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828356.515372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515387:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb222300 x1874663333109376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515400:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333109376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515410:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515413:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515417:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828356.515423:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828356.515441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515451:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb220380 x1874663333109504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515549:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333109504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515560:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515564:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515568:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333109504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828356.515607:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828356.515635:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515649:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb220700 x1874663333109632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515673:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333109632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515686:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515690:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828356.515696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828356.515714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515745:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb221880 x1874663333109760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515751:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333109760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828356.515768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828356.515778:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828356.515784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828356/real 1787828356] req@ffff9057fb223100 x1874663333109888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828356.515790:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333109888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828356.515795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828356.515797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828356.515814:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333109888:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828356.898513:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828357.922537:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828358.946621:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828359.970472:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828360.994374:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828361.634298:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634306:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828361.634310:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634318:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634327:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634344:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828361.634375:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333110016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634396:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634398:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828361.634400:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634402:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634405:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634407:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828361.634414:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828361.634423:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333110144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634430:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828361.634434:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634436:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634438:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634443:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634445:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333110016:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828361.634447:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828361.634457:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333110272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828361.634468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634481:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634484:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634486:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828361.634490:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828361.634497:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333110400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828361.634507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634509:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634511:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634514:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828361.634516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828361.634521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828361.634525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333110528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828361.634537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634540:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634542:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634545:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828361.634550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828361.634557:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb223100 x1874663333110016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828361.634560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333110656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828361.634592:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333110016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828361.634592:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828361.634609:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828361.634611:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828361.634613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828361.634617:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.634623:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333110016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828361.634625:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333110784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828361.634642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828361.634646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828361.634647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828361.634650:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10172 00000100:00080000:3.0:1787828361.634652:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828361.634653:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110144:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828361.634657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828361.634662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828361.634672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333110912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828361.634679:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634692:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb221880 x1874663333110144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.634714:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333110144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.634720:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.634722:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.634724:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828361.634772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828361.634799:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634810:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb220700 x1874663333110272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.634820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333110272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.634828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.634830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.634833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828361.634838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828361.634853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb220380 x1874663333110400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.634871:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333110400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.634878:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.634880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.634882:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828361.634887:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828361.634900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634908:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb222300 x1874663333110528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.634917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333110528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.634925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.634927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.634929:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333110528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828361.634934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828361.634947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.634973:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb221c00 x1874663333110656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.634988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333110656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.635016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.635019:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.635023:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828361.635030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828361.635051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.635069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb222d80 x1874663333110784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.635082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333110784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.635107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.635109:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.635111:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828361.635115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828361.635128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828361.635138:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff9057fb222a00 x1874663333110912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828361.635146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333110912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828361.635152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828361.635153:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828361.635172:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333110912:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828362.018465:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828363.042486:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828364.066491:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828365.090489:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828366.114527:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828366.754366:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828366.754382:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754394:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754406:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754415:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828366.754469:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333111040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754499:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828366.754504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754508:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754511:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828366.754523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828366.754533:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333111168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754542:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754545:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.754547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754551:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754556:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754560:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828366.754563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828366.754573:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333111296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828366.754588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333111040:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828366.754594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754622:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828366.754632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828366.754645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333111424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754653:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.754658:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754662:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754665:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754668:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828366.754673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828366.754674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828366.754689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333111552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754698:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828366.754703:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754708:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:1.0:1787828366.754712:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828366.754730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78aaa00 x1874663333111040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828366.754735:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828366.754744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828366.754782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333111680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828366.754782:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333111040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828366.754859:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.754862:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.754870:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828366.754874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828366.754876:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00080000:2.0:1787828366.754885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828366.754886:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754901:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828366.754902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333111040:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828366.754912:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828366.754920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828366.754930:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333111808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.754942:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828366.754951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828366.754957:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.754960:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.754968:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10177 00000100:00100000:2.0:1787828366.754974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78a8a80 x1874663333111168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828366.754975:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.754987:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755006:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333111168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828366.755008:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828366.755025:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333111936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828366.755027:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.755096:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828366.755117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78ab100 x1874663333111296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755134:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333111296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755147:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.755152:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828366.755163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78a9500 x1874663333111424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755176:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333111424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828366.755188:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828366.755198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755203:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78a8700 x1874663333111552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755270:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333111552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755278:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755280:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333111552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828366.755284:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828366.755300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755308:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78a9c00 x1874663333111680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755315:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333111680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755322:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755324:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.755327:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828366.755338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755344:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78aa300 x1874663333111808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755351:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333111808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755362:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755365:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.755368:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828366.755378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828366.755383:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff9057c78a9180 x1874663333111936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.755390:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333111936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.755396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.755397:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828366.755399:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333111936:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828367.138477:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828368.162405:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828369.186440:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828370.210486:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828371.234530:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828371.874337:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874347:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828371.874351:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:3.0:1787828371.874374:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828371.874423:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333112064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828371.874451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828371.874457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874462:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:3.0:1787828371.874467:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874473:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828371.874497:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333112064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828371.874504:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333112192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828371.874517:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874520:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828371.874522:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874527:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:3.0:1787828371.874531:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874534:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874539:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828371.874550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333112320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828371.874559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828371.874568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828371.874568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:3.0:1787828371.874588:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874594:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828371.874615:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb223800 x1874663333112064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828371.874622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333112448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828371.874646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333112064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828371.874646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828371.874671:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828371.874676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828371.874680:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:2.0:1787828371.874685:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828371.874686:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874696:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828371.874702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333112064:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828371.874704:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828371.874724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333112576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828371.874733:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874737:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828371.874739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112192:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828371.874740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874754:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:3.0:1787828371.874759:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874763:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874768:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828371.874775:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828371.874786:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333112704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828371.874795:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb222680 x1874663333112192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.874810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333112192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828371.874811:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874829:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828371.874836:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828371.874838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828371.874845:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00080000:2.0:1787828371.874849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828371.874854:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828371.874860:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112192:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828371.874862:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874877:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828371.874880:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828371.874893:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333112832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828371.874906:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828371.874907:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828371.874911:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828371.874914:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828371.874920:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10182 00000100:00100000:2.0:1787828371.874922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb220e00 x1874663333112320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828371.874925:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828371.874937:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828371.874947:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828371.874951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333112320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.874963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828371.874964:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333112960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828371.874971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.874984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828371.875005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828371.875033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828371.875041:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb220a80 x1874663333112448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.875049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333112448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.875054:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.875056:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.875057:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828371.875061:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828371.875072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828371.875077:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb221180 x1874663333112576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.875084:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333112576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.875090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.875092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.875093:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333112576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828371.875105:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828371.875116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828371.875122:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb220000 x1874663333112704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.875129:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333112704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.875138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.875140:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.875143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828371.875146:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828371.875156:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828371.875161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb223b80 x1874663333112832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.875168:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333112832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.875187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.875189:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.875191:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828371.875196:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828371.875208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828371.875215:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff9057fb221500 x1874663333112960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.875258:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333112960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.875264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.875266:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828371.875269:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333112960:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828372.258567:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828373.282515:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828374.306466:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828375.330479:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828376.354436:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828376.994299:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994309:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828376.994313:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994324:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994353:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994361:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828376.994403:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333113088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828376.994428:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828376.994435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994449:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828376.994478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333113216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828376.994493:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828376.994505:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333113088:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828376.994506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994524:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994528:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828376.994550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333113344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828376.994559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828376.994564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994578:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994593:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828376.994603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333113472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828376.994612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828376.994617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994620:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994623:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994627:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828376.994633:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113216:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828376.994652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828376.994668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333113600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828376.994677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113344:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828376.994681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994688:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828376.994691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994712:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00100000:2.0:1787828376.994714:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113472:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828376.994725:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994731:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828376.994754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333113728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828376.994755:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113600:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828376.994778:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828376.994781:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828376.994784:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994788:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994791:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994795:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994799:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828376.994810:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333113856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828376.994819:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828376.994820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113728:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828376.994835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828376.994838:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828376.994845:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10187 00000100:00080000:3.0:1787828376.994853:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828376.994857:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828376.994861:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828376.994866:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828376.994875:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333113984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828376.994898:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113984:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828377.378412:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828378.402385:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828379.426311:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828380.066557:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828380.066630:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828380.066656:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066697:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066709:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066716:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066723:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066731:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066741:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828380.066753:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828380.066761:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb220e00 x1874663333113728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333113728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.066802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.066806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.066810:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828380.066816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb220a80 x1874663333113600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066856:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333113600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.066866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.066870:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.066873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828380.066881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb223b80 x1874663333113216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333113216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.066901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.066905:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.066909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828380.066916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb220000 x1874663333113344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066927:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333113344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.066935:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.066939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.066942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828380.066949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb221180 x1874663333113472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333113472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.066971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.066975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.066978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333113472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828380.066985:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb221500 x1874663333113088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.066999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333113088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.067009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.067013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.067017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333113088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828380.067051:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb223800 x1874663333113984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.067067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333113984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.067076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.067081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.067084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828380.067092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828380] req@ffff9057fb222680 x1874663333113856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828380.067103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333113856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828380.067132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828380.067138:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828380.067142:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333113856:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1787828380.450529:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828381.474444:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828382.114289:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114298:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828382.114301:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114309:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114326:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114332:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828382.114364:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333114112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.114388:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114391:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.114392:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114395:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114398:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114401:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114409:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.114418:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333114240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.114426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114429:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.114430:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828382.114435:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333114112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828382.114436:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114441:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114447:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.114459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333114368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.114466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.114470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114473:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114475:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114477:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114481:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828382.114488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333114496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.114503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828382.114509:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828382.114510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.114515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114535:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114541:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114547:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828382.114549:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb222a00 x1874663333114112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828382.114564:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333114624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.114575:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333114112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.114577:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.114584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114592:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114600:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828382.114602:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.114609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828382.114613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828382.114619:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333114112:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828382.114623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.114640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333114752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.114653:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114240:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828382.114653:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.114666:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114672:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114677:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114682:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114688:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828382.114695:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828382.114715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333114880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.114716:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb222d80 x1874663333114240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.114748:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.114754:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828382.114755:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333114240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.114757:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.114767:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10192 00000100:00080000:3.0:1787828382.114774:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.114782:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828382.114793:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828382.114795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.114802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.114807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828382.114812:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333115008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.114820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.114855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.114871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb221c00 x1874663333114368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.114884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333114368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.114892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.114895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.114899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.114905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.114928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.114941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb222300 x1874663333114496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.114955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333114496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.114964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.114968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.114972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828382.114979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.115001:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.115013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb220380 x1874663333114624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.115026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333114624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.115034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.115037:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.115040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333114624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828382.115063:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333114752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.115085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.115098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb220700 x1874663333114752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.115132:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333114752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.115141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.115144:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.115147:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333114752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.115153:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333114880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.115176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.115188:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb221880 x1874663333114880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.115200:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333114880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.115210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.115213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.115216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333114880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.115261:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.115286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828382.115301:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9057fb223100 x1874663333115008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.115315:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333115008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.115324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.115328:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828382.115331:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115008:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828382.498527:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828383.522398:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828384.546450:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828385.570318:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828386.594466:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828387.234392:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234407:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828387.234414:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234429:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234447:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.234458:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828387.234537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333115136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.234594:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828387.234603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234609:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.234626:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.234646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828387.234682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333115136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828387.234685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333115264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.234710:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234716:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828387.234719:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234725:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234729:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.234732:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.234739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828387.234754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333115392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.234761:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234765:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828387.234766:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234770:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234773:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828387.234782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828387.234788:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.234795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828387.234808:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333115520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.234816:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234819:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828387.234821:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234824:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234829:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.234832:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.234835:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828387.234844:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333115648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828387.234855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb223100 x1874663333115136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828387.234855:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.234868:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828387.234874:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828387.234889:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.234901:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.234916:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.234953:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828387.234970:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333115136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828387.234972:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333115776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.235105:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828387.235109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828387.235133:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828387.235137:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828387.235139:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828387.235146:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333115136:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828387.235147:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.235160:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.235167:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828387.235191:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828387.235209:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828387.235215:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333115904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828387.235241:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828387.235245:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828387.235247:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828387.235318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828387.235320:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10197 00000100:00080000:3.0:1787828387.235336:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828387.235347:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235360:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb221880 x1874663333115264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828387.235361:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828387.235396:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333115264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828387.235410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333116032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828387.235436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235460:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235473:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828387.235486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828387.235511:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235532:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb220700 x1874663333115392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235546:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333115392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235555:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235559:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235562:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828387.235571:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828387.235590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235600:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb220380 x1874663333115520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235613:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333115520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235621:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235625:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235628:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828387.235634:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828387.235650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235660:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb222300 x1874663333115648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333115648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235683:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235686:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333115648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828387.235692:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828387.235709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235719:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb221c00 x1874663333115776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333115776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235742:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235746:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828387.235767:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828387.235785:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235795:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb222d80 x1874663333115904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235808:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333115904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235822:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333115904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828387.235845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828387.235876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828387.235905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828387/real 1787828387] req@ffff9057fb222a00 x1874663333116032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828387.235920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333116032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828387.235956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828387.235960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828387.235964:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116032:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828387.618598:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828388.642541:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828389.666475:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828390.690520:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828391.714469:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828392.354343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354355:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828392.354360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354395:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828392.354452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333116160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828392.354500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828392.354507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354513:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354516:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354532:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828392.354546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333116288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828392.354560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828392.354568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354573:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354578:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354582:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828392.354603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333116416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828392.354613:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333116160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828392.354625:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828392.354634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354640:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354645:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354649:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354656:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828392.354673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333116544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828392.354685:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354690:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828392.354692:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354696:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828392.354708:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828392.354715:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828392.354741:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333116672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828392.354755:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354759:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828392.354762:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828392.354770:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb222680 x1874663333116160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828392.354771:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354783:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354795:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828392.354817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333116160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828392.354826:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333116800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828392.354845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.354860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.354875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333116160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828392.354881:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354888:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828392.354893:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116288:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828392.354895:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828392.354912:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.354918:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828392.354923:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828392.354930:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828392.354939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828392.354951:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333116928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828392.354963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb223800 x1874663333116288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828392.354977:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828392.354990:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828392.354993:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828392.354998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333116288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828392.355005:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10203 00000100:00080000:3.0:1787828392.355018:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828392.355026:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828392.355052:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828392.355064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828392.355065:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828392.355076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828392.355090:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333117056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828392.355117:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828392.355147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb221500 x1874663333116416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355176:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333116416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355194:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828392.355201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828392.355259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355275:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb221180 x1874663333116544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355290:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333116544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355304:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355308:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828392.355315:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828392.355338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355350:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb220000 x1874663333116672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355364:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333116672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355382:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355388:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333116672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828392.355399:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828392.355436:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355460:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb223b80 x1874663333116800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355485:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333116800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355507:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355513:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828392.355526:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828392.355566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355588:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb220a80 x1874663333116928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355606:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333116928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355625:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355629:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333116928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828392.355641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828392.355667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828392.355679:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828392/real 1787828392] req@ffff9057fb220e00 x1874663333117056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828392.355693:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333117056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828392.355703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828392.355706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828392.355710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117056:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828392.738461:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828393.762490:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828394.786563:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828395.810491:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828396.834514:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828397.474456:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.474469:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828397.474475:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.474488:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.474502:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.474512:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828397.474561:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333117184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.474603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.474608:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828397.474610:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.474615:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.474620:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.474624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828397.474635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828397.474673:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333117312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.474688:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.474692:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828397.474695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.474700:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.474704:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.474715:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828397.474727:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333117184:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828397.474729:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828397.474760:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333117440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.474783:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.474791:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828397.474805:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.474813:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.474823:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.474832:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828397.474842:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828397.474846:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828397.474885:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333117568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.474900:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.474904:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828397.474907:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828397.474915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158e680 x1874663333117184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828397.474917:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.474929:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.474951:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828397.474964:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828397.474972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333117184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.474990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475025:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828397.475027:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333117696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828397.475038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333117184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828397.475053:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.475057:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828397.475060:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.475076:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.475084:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828397.475086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117312:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828397.475097:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828397.475108:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828397.475141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828397.475147:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333117824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.475165:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828397.475170:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158f800 x1874663333117312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828397.475172:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828397.475179:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.475192:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00080000:0.0:1787828397.475207:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828397.475215:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333117312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828397.475217:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828397.475280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828397.475290:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828397.475292:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475304:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828397.475319:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333117952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828397.475342:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828397.475346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828397.475349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828397.475361:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10208 00000100:00100000:2.0:1787828397.475362:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117440:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828397.475373:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828397.475381:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828397.475395:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828397.475414:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333118080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828397.475423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475474:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158c000 x1874663333117440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.475503:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333117440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.475543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475548:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475553:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828397.475562:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828397.475593:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475609:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158dc00 x1874663333117568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.475624:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333117568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.475634:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475637:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475641:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828397.475648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828397.475668:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475681:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158e300 x1874663333117696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.475705:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333117696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.475730:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475736:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475743:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333117696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828397.475754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828397.475792:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475810:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158f100 x1874663333117824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.475824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333117824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.475835:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475843:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828397.475850:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828397.475880:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158ca80 x1874663333117952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.475904:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333117952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.475914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.475917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.475925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333117952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828397.475948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828397.475968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828397.475981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828397/real 1787828397] req@ffff9057f158df80 x1874663333118080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828397.476028:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333118080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828397.476055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828397.476061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828397.476069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118080:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828397.858509:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828398.882356:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828399.906459:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828400.930448:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828401.954511:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828402.594390:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594403:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828402.594407:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594419:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.594431:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594440:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828402.594487:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333118208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.594542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828402.594549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594554:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.594558:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594575:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828402.594593:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333118336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.594606:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594611:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.594613:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594618:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00100000:2.0:1787828402.594626:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333118208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828402.594627:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594640:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594646:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828402.594661:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333118464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.594674:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594677:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828402.594679:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594684:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.594689:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594691:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594696:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828402.594704:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333118592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828402.594715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828402.594717:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594721:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.594722:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594726:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.594728:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594732:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594735:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828402.594743:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333118720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.594750:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828402.594755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594757:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.594761:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594765:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594770:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828402.594776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158df80 x1874663333118208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828402.594782:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333118848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.594865:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594870:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.594872:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.594879:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00100000:2.0:1787828402.594882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333118208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828402.594884:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.594902:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828402.594917:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828402.594919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.594933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828402.594938:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333118976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828402.594939:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333118208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828402.594980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118336:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828402.594988:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.594993:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.595023:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.595030:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10213 00000100:00080000:0.0:1787828402.595037:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.595048:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828402.595050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828402.595063:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828402.595074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158ca80 x1874663333118336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828402.595082:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333119104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828402.595132:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333118336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595219:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595223:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828402.595284:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828402.595300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595309:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158f100 x1874663333118464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595317:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333118464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595325:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595327:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828402.595332:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828402.595343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595349:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158e300 x1874663333118592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595357:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333118592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595363:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595365:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828402.595369:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828402.595379:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595385:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158dc00 x1874663333118720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595392:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333118720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595400:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333118720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828402.595406:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828402.595416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595422:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158c000 x1874663333118848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595429:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333118848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595436:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595438:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828402.595441:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828402.595451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595457:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158f800 x1874663333118976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595464:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333118976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595471:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595473:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333118976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828402.595477:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333119104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828402.595488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828402.595493:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff9057f158e680 x1874663333119104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828402.595501:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333119104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828402.595506:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828402.595508:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828402.595510:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333119104:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828402.978571:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828404.002499:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828405.026470:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828406.050472:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828407.074458:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828407.714394:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.714418:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828407.714424:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.714436:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.714450:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.714461:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828407.714528:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333119232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828407.714578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.714586:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828407.714590:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.714599:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.714621:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.714644:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828407.714658:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333119232:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828407.714661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828407.714693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333119360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828407.714719:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.714724:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828407.714726:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.714732:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.714738:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.714743:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828407.714750:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828407.714766:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333119488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828407.714782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828407.714793:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.714803:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828407.714808:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.714819:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.714827:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.714836:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828407.714846:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb220e00 x1874663333119232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828407.714848:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828407.714887:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333119616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828407.714888:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333119232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.714969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828407.714971:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.714983:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828407.714984:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828407.714988:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828407.714997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333119232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828407.715001:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.715017:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.715028:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828407.715036:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828407.715052:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828407.715059:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333119744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828407.715097:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.715103:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828407.715106:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.715113:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00080000:3.0:1787828407.715120:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.715126:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828407.715133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828407.715135:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828407.715166:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333119872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828407.715175:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb220a80 x1874663333119360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828407.715193:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.715208:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828407.715212:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.715273:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10218 00000100:00100000:2.0:1787828407.715277:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333119360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828407.715278:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828407.715290:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828407.715303:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828407.715309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715319:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828407.715326:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333120000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828407.715333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828407.715356:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119488:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828407.715369:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828407.715379:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828407.715384:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828407.715396:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10218 00000800:00000100:2.0:1787828407.715398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828407.715419:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828407.715430:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb223b80 x1874663333119488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828407.715433:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828407.715450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828407.715483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333120128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828407.715489:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333119488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715537:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715544:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828407.715553:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828407.715581:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828407.715594:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb220000 x1874663333119616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828407.715617:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333119616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715627:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715631:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715634:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828407.715655:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828407.715686:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828407.715697:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb221180 x1874663333119744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828407.715711:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333119744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715733:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715736:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333119744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828407.715746:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333119872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828407.715765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828407.715776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb221500 x1874663333119872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828407.715790:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333119872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715808:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715811:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333119872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828407.715822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828407.715840:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828407.715850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb223800 x1874663333120000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828407.715863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333120000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828407.715886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828407.715904:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828407.715914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff9057fb222680 x1874663333120128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828407.715952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333120128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828407.715963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828407.715966:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828407.715971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120128:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828408.098476:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828409.122482:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828410.146460:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828411.170451:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828412.194496:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828412.834313:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834321:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828412.834326:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834357:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834368:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834375:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828412.834415:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333120256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828412.834461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828412.834467:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834475:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834478:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828412.834513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333120384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828412.834525:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828412.834531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834535:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834537:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834542:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828412.834550:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333120256:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828412.834551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828412.834568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333120512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828412.834586:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828412.834591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834594:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834598:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834601:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828412.834615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333120640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828412.834625:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828412.834631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834634:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834641:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828412.834655:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828412.834659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333120768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828412.834674:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834678:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828412.834680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834686:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834694:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828412.834698:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834703:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828412.834716:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333120896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828412.834723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb222a00 x1874663333120256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828412.834731:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834743:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828412.834748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828412.834761:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834766:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828412.834783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333120256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828412.834783:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834793:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828412.834823:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828412.834824:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333121024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828412.834833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.834844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333120256:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828412.834845:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828412.834854:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828412.834856:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828412.834860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828412.834864:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10223 00000100:00080000:3.0:1787828412.834910:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828412.834913:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828412.834918:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828412.834925:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828412.834936:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb222d80 x1874663333120384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828412.834940:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333121152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828412.834988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333120384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835002:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835006:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835010:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828412.835035:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828412.835052:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835072:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb221c00 x1874663333120512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333120512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828412.835099:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828412.835111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835119:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb222300 x1874663333120640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333120640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835145:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835147:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828412.835152:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828412.835163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835170:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb220380 x1874663333120768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835179:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333120768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835188:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835191:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333120768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828412.835195:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828412.835206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835213:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb220700 x1874663333120896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835252:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333120896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835261:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835264:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333120896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828412.835279:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828412.835304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835313:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb221880 x1874663333121024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835323:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333121024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835331:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835335:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828412.835339:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828412.835361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828412.835368:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff9057fb223100 x1874663333121152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828412.835377:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333121152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828412.835383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828412.835385:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828412.835388:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121152:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828413.218546:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828414.242402:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828415.266524:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828416.290559:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828417.314589:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828417.954401:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954413:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828417.954418:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954431:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954444:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954453:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828417.954510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333121280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828417.954558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954562:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954564:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954583:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828417.954606:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333121408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954619:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828417.954621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828417.954643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333121536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954670:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828417.954675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954681:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954685:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954688:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954694:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828417.954706:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333121664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828417.954709:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333121280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828417.954739:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954748:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828417.954750:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954755:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954757:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954760:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954764:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828417.954778:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333121792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954786:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954789:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828417.954791:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954796:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954800:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954805:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954809:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828417.954818:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333121920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954829:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954832:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828417.954839:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828417.954840:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828417.954870:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954877:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828417.954910:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954918:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828417.954935:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333122048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828417.954948:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828417.954952:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828417.954955:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828417.954960:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10228 00000100:00080000:1.0:1787828417.954964:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828417.954968:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78aa680 x1874663333121280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828417.954974:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828417.954991:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828417.955015:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333121280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828417.955023:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333122176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828417.955065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333121280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828417.955094:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828417.955127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78a9180 x1874663333121408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955182:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333121408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828417.955208:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828417.955229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955241:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78aa300 x1874663333121536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955314:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333121536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955326:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955329:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828417.955356:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828417.955383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955400:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78a9c00 x1874663333121664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955430:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333121664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955448:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955451:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828417.955469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828417.955490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955522:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78a8700 x1874663333121792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955536:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333121792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955547:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955551:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955554:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333121792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828417.955561:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828417.955582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955598:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78a9500 x1874663333121920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955621:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333121920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333121920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828417.955686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828417.955711:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955742:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78ab100 x1874663333122048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333122048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955771:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955775:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828417.955782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828417.955804:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828417.955815:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828417] req@ffff9057c78a8a80 x1874663333122176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828417.955828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333122176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828417.955838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828417.955841:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828417.955846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122176:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828418.338552:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828419.362428:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828420.386509:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828421.410445:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828422.434449:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828423.074359:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.074370:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828423.074375:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.074384:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.074395:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.074403:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828423.074445:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333122304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.074480:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.074486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.074489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.074496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.074503:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.074511:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828423.074525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828423.074547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333122304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828423.074551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333122432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.074574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.074580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.074583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.074590:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.074611:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.074622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828423.074624:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828423.074637:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828423.074661:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333122560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.074679:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828423.074681:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb223100 x1874663333122304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828423.074692:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.074698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.074720:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.074727:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828423.074735:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333122304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828423.074739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828423.074753:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828423.074761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.074772:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828423.074778:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333122688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828423.074785:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333122304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828423.074807:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.074813:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.074839:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828423.074842:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122432:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828423.074851:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.074860:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.074878:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828423.074885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828423.074893:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828423.074905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb221880 x1874663333122432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828423.074920:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333122816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828423.074923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333122432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.074968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.074975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828423.074978:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828423.074982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122432:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828423.074990:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.074997:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828423.074998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122560:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828423.075013:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.075025:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828423.075028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828423.075037:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828423.075045:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb220700 x1874663333122560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828423.075049:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828423.075065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333122560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828423.075078:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333122944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828423.075083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828423.075092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828423.075107:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828423.075128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828423.075130:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.075138:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828423.075141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb220380 x1874663333122688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828423.075144:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828423.075156:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333122688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.075167:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828423.075170:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122688:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828423.075190:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00100000:2.0:1787828423.075191:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122816:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828423.075204:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.075216:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828423.075256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828423.075264:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828423.075278:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb222300 x1874663333122816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828423.075292:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333123072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828423.075296:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333122816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.075344:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828423.075345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828423.075353:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333122816:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828423.075355:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.075362:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.075371:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10233 00000100:00080000:3.0:1787828423.075378:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828423.075382:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122944:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828423.075391:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828423.075402:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828423.075413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828423.075428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333123200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828423.075431:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb221c00 x1874663333122944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828423.075510:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333122944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075523:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.075527:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828423.075531:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333122944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828423.075554:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828423.075578:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828423.075590:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb222d80 x1874663333123072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828423.075603:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333123072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075613:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.075616:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828423.075621:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828423.075628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828423.075672:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828423.075695:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9057fb222a00 x1874663333123200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828423.075741:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333123200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828423.075753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828423.075758:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828423.075763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123200:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828423.458486:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828424.482511:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828425.506501:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828426.530439:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828427.554498:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828428.194324:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828428.194344:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.194356:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194370:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.194379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828428.194427:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333123328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.194462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.194470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.194475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194479:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.194483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828428.194495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828428.194512:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333123456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.194527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194533:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.194537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828428.194543:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333123328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828428.194544:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194553:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.194561:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828428.194566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828428.194583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333123584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.194595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194599:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.194602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.194609:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194631:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828428.194637:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828428.194648:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828428.194661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828428.194689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333123712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828428.194708:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb222680 x1874663333123328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.194724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194735:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.194738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828428.194746:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333123328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.194747:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194759:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.194773:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828428.194805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828428.194813:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828428.194817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.194837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333123328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828428.194847:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333123840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.194861:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.194867:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.194871:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828428.194873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123456:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828428.194881:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.194893:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.194924:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828428.194936:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828428.194937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828428.194962:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333123968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828428.194968:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb223800 x1874663333123456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.194992:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.195017:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.195023:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828428.195024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333123456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.195036:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.195047:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.195056:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828428.195057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828428.195063:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828428.195070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195077:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828428.195092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828428.195092:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333124096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.195130:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828428.195132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828428.195144:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.195150:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828428.195156:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb221500 x1874663333123584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.195166:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10238 00000100:00080000:3.0:1787828428.195185:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828428.195194:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333123584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.195200:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828428.195216:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828428.195253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195261:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195271:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828428.195282:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828428.195291:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333124224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828428.195322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828428.195345:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb221180 x1874663333123712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828428.195370:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333123712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828428.195380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195384:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195388:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828428.195394:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828428.195413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828428.195424:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb220000 x1874663333123840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828428.195437:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333123840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828428.195447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195451:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195454:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333123840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828428.195475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828428.195494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828428.195505:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb223b80 x1874663333123968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828428.195519:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333123968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828428.195537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195540:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195544:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333123968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828428.195551:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828428.195569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828428.195580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb220a80 x1874663333124096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828428.195593:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333124096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828428.195603:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195606:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195610:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828428.195616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828428.195634:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828428.195644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9057fb220e00 x1874663333124224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828428.195657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333124224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828428.195667:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828428.195670:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828428.195675:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124224:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828428.578472:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828429.602518:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828430.626524:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828431.650472:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828432.674415:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828433.314405:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.314421:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828433.314428:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314438:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314451:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828433.314459:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828433.314502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333124352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.314534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.314538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828433.314540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314545:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314549:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828433.314553:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.314563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828433.314577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333124480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.314589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.314593:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828433.314595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314599:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314602:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828433.314606:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.314614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828433.314614:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333124352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828433.314634:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333124608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.314646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.314649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828433.314652:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314656:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314661:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828433.314664:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.314670:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828433.314684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333124736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.314711:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828433.314712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828433.314722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828433.314727:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314736:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314760:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828433.314768:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.314776:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828433.314805:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158d880 x1874663333124352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828433.314811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333124864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.314837:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.314842:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828433.314843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333124352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828433.314847:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314861:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:2.0:1787828433.314862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.314871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828433.314872:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828433.314877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333124352:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828433.314885:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828433.314889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124480:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828433.314896:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828433.314919:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333124992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828433.314920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828433.314934:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828433.314937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158d180 x1874663333124480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828433.314939:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828433.314943:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.314954:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.314962:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828433.314966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333124480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828433.314971:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.314981:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828433.314983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.314989:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.314992:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828433.314998:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333125120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828433.315009:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828433.315012:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828433.315014:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828433.315018:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10244 00000100:00080000:0.0:1787828433.315021:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828433.315022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124608:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828433.315027:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828433.315034:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828433.315047:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333125248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828433.315053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158d500 x1874663333124608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333124608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315094:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828433.315101:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828433.315114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315122:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158ed80 x1874663333124736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315130:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333124736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315139:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828433.315145:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828433.315157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158fb80 x1874663333124864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315173:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333124864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315182:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333124864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828433.315188:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828433.315200:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315207:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158ea00 x1874663333124992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315216:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333124992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315224:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315227:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333124992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828433.315231:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333125120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828433.315274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315284:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158ce00 x1874663333125120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315293:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333125120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315302:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315305:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333125120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828433.315321:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333125248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828433.315334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828433.315342:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828433/real 1787828433] req@ffff9057f158c700 x1874663333125248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828433.315351:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333125248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828433.315357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828433.315359:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828433.315361:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333125248:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828433.698485:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828434.722399:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828435.746446:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828436.770416:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828437.794498:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828438.434428:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.434452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828438.434459:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.434480:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.434526:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.434541:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828438.434614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333125376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828438.434668:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.434674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828438.434677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.434695:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.434702:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.434732:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.434746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828438.434780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333125504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828438.434798:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.434807:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828438.434809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333125376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828438.434811:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.434826:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.434834:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.434839:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.434846:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828438.434865:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333125632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828438.434879:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.434884:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828438.434887:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.434892:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.434922:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.434927:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.434933:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828438.434946:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828438.434954:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333125760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828438.434969:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.434974:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828438.434977:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.434982:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.434990:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.434994:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.435000:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828438.435016:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333125888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828438.435037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb220e00 x1874663333125376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828438.435044:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.435059:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828438.435065:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.435078:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.435083:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.435089:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.435096:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828438.435098:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333125376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828438.435115:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333126016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828438.435118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828438.435142:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828438.435149:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333125376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828438.435151:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828438.435158:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.435170:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.435174:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.435179:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.435185:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828438.435198:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333126144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828438.435204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125504:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828438.435304:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828438.435313:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828438.435323:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828438.435333:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10249 00000100:00080000:3.0:1787828438.435341:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828438.435349:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828438.435357:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828438.435378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828438.435386:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333126272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828438.435408:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb220a80 x1874663333125504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435434:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333125504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435451:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435455:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828438.435463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828438.435488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435501:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb223b80 x1874663333125632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435515:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333125632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435525:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435528:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435532:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828438.435539:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828438.435561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435572:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb220000 x1874663333125760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435595:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333125760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435608:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435611:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828438.435618:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828438.435637:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435648:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb221180 x1874663333125888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435660:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333125888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435673:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333125888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828438.435683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828438.435702:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435742:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb221500 x1874663333126016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435758:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333126016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435777:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435781:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435785:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828438.435812:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828438.435838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435851:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb223800 x1874663333126144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333126144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.435883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.435890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.435896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828438.435909:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828438.435947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828438.435970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828438/real 1787828438] req@ffff9057fb222680 x1874663333126272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828438.435989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333126272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828438.436000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828438.436004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828438.436009:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333126272:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828438.818475:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828439.842436:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828440.866462:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828441.890521:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828442.914412:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828443.554361:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554369:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828443.554372:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554380:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554389:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554395:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828443.554431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333126400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554456:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554459:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828443.554460:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554463:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554475:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828443.554483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333126528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828443.554495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554499:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554501:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828443.554512:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333126656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828443.554517:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333126400:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828443.554521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828443.554527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554531:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554533:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554535:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554539:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828443.554547:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333126784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828443.554557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554559:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828443.554574:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333126912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828443.554583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554586:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554588:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554590:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554593:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828443.554599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333127040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828443.554616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554618:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554621:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554626:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828443.554628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126528:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828443.554631:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828443.554646:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333127168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828443.554653:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828443.554656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828443.554657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828443.554659:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10254 00000100:00080000:0.0:1787828443.554662:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828443.554664:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828443.554668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828443.554695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333127296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828443.554696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828443.554733:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828443.554760:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828443.554782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828443.554804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828443.554829:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127296:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828443.938466:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828444.962769:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828445.986465:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828446.626516:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1787828446.626556:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828446.626574:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626632:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626642:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626663:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626674:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626682:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828446.626692:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828446.626698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158ea00 x1874663333126656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787828446.626707:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828446.626731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333126656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626745:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828446.626754:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158fb80 x1874663333126784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626767:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333126784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626775:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828446.626781:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158ed80 x1874663333126912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333126912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626808:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626811:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828446.626821:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158d500 x1874663333127040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626833:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333127040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626845:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828446.626856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158d180 x1874663333127168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626870:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333127168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626878:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626881:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828446.626893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158c700 x1874663333126400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333126400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626919:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626923:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333126400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828446.626930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158ce00 x1874663333126528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.626944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333126528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.626953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.626957:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.626961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333126528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828446.626997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828443/real 1787828446] req@ffff9057f158d880 x1874663333127296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828446.627010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333127296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828446.627017:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828446.627022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828446.627025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333127296:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787828447.010384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828448.034451:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828448.674426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.674438:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828448.674443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.674454:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.674482:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.674489:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828448.674545:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333127424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828448.674588:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.674602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828448.674607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.674615:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.674621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.674629:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.674670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828448.674706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333127424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828448.674716:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333127552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828448.674748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.674754:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828448.674757:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.674764:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.674782:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.674788:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.674797:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828448.674808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828448.674817:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333127680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828448.674834:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.674848:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828448.674851:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.674858:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.674864:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.674873:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828448.674876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78aaa00 x1874663333127424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828448.674885:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828448.674911:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333127808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828448.674918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333127424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.674945:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.674953:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828448.674956:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828448.674958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828448.674966:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:2.0:1787828448.674972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828448.674975:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828448.674986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333127424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828448.674987:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.675000:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828448.675021:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333127936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828448.675036:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828448.675037:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.675051:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828448.675053:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.675061:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.675066:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.675073:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.675080:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828448.675089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828448.675102:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333128064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828448.675116:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78a9f80 x1874663333127552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.675120:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.675134:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828448.675139:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.675150:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00100000:2.0:1787828448.675157:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333127552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.675160:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.675173:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.675185:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828448.675200:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675208:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828448.675214:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333128192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828448.675220:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828448.675275:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127680:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828448.675299:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828448.675304:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828448.675308:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828448.675316:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10259 00000100:00080000:1.0:1787828448.675322:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828448.675324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828448.675334:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828448.675343:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828448.675366:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78a8380 x1874663333127680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828448.675367:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333128320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828448.675419:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333127680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675440:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675446:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828448.675457:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828448.675494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828448.675515:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78a9880 x1874663333127808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828448.675533:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333127808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675545:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675549:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675555:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828448.675580:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828448.675611:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828448.675629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78ab800 x1874663333127936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828448.675663:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333127936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675681:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675686:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333127936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828448.675696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828448.675724:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828448.675740:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78abb80 x1874663333128064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828448.675756:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333128064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675772:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675776:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828448.675784:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828448.675809:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828448.675825:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78a8000 x1874663333128192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828448.675844:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333128192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828448.675873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828448.675900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828448.675917:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff9057c78aad80 x1874663333128320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828448.675936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333128320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828448.675948:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828448.675952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828448.675957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333128320:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828449.058332:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828450.082541:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828451.106421:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828452.130671:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828453.154487:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828453.794380:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794399:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828453.794406:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794436:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794455:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.794465:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828453.794527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333128448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.794576:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794581:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828453.794584:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794594:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.794599:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828453.794649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828453.794672:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333128576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.794688:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794692:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828453.794695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794701:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794706:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.794716:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828453.794720:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333128448:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828453.794724:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828453.794747:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333128704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.794761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794766:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828453.794768:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794773:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794778:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.794783:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828453.794789:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828453.794804:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333128832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.794815:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794819:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828453.794831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794835:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794841:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828453.794849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828453.794853:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828453.794862:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828453.794877:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333128960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.794911:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.794916:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828453.794918:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.794925:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.794931:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.794942:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828453.794942:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158e680 x1874663333128448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828453.794954:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828453.794969:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333129088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.795033:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.795039:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828453.795042:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.795047:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00080000:0.0:1787828453.795053:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828453.795061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333128448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828453.795063:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828453.795074:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828453.795094:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333129216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828453.795114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828453.795117:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.795133:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828453.795135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828453.795141:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.795157:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10264 00000100:00100000:2.0:1787828453.795171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333128448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828453.795176:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.795194:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828453.795205:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128576:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828453.795210:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828453.795289:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333129344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828453.795349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.795384:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158f800 x1874663333128576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.795414:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333128576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.795434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.795441:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.795449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828453.795503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828453.795559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.795584:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158c000 x1874663333128704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.795641:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333128704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.795671:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.795679:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.795686:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828453.795699:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828453.795743:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.795769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158dc00 x1874663333128832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.795795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333128832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.795812:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.795818:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.795826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828453.795850:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828453.795888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.795924:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158e300 x1874663333128960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.795949:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333128960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.795965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.795972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.795979:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333128960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828453.795991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828453.796028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.796049:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158f100 x1874663333129088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.796074:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333129088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.796090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.796104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.796112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828453.796124:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828453.796163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.796194:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158ca80 x1874663333129216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.796219:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333129216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.796276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.796284:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.796291:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828453.796305:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828453.796349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828453.796375:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff9057f158df80 x1874663333129344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828453.796401:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333129344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828453.796431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828453.796437:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828453.796445:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333129344:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828454.178457:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828455.202458:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828456.226500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828457.250418:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828458.274440:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828458.914324:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914335:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828458.914340:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914351:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914372:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828458.914425:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333129472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914456:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914460:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828458.914462:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914470:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914473:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828458.914499:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333129600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914512:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914516:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.914518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828458.914526:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333129472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828458.914528:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914538:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914544:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828458.914563:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333129728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914599:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828458.914602:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914607:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914611:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914614:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828458.914632:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333129856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914645:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.914647:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914650:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914653:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914656:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914659:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828458.914668:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333129984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914677:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914681:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828458.914682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914688:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914694:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828458.914696:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828458.914701:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914708:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828458.914720:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333130112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.914731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914735:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.914736:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.914740:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:0.0:1787828458.914745:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.914749:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914754:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828458.914764:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158df80 x1874663333129472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828458.914770:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333130240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828458.914802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333129472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828458.914808:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.914821:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.914826:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828458.914832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828458.914836:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10269 00000100:00080000:2.0:1787828458.914843:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828458.914844:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828458.914850:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333129472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828458.914854:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828458.914863:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828458.914877:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333130368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828458.914888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828458.914920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.914940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158ca80 x1874663333129600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.914957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333129600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.914968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.914972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.914978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828458.914988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828458.915012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158f100 x1874663333129728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915038:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333129728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828458.915057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828458.915073:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915082:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158e300 x1874663333129856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915108:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333129856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915120:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915123:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828458.915128:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828458.915146:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915156:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158dc00 x1874663333129984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915168:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333129984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915178:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915181:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333129984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828458.915186:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828458.915201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915211:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158c000 x1874663333130112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915253:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333130112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915266:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915270:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828458.915276:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828458.915315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915333:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158f800 x1874663333130240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915353:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333130240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915373:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915379:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828458.915407:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828458.915431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828458.915443:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff9057f158e680 x1874663333130368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828458.915454:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333130368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828458.915463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828458.915466:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828458.915469:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333130368:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828459.298610:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828460.322538:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828461.346386:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828462.370390:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828463.394375:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828464.034370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828464.034382:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034390:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034399:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.034405:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828464.034441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333130496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.034464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034469:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.034470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034478:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.034483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034492:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.034506:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333130624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.034511:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333130496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828464.034518:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034521:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.034523:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034526:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034543:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.034548:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.034591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333130752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.034601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.034605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034609:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034612:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828464.034616:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828464.034616:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828464.034642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333130880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.034651:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034654:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.034656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034660:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034663:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828464.034666:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb222a00 x1874663333130496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.034667:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828464.034682:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333131008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.034685:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333130496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.034693:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828464.034696:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828464.034697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.034699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828464.034701:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828464.034703:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00100000:2.0:1787828464.034705:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333130496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828464.034707:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.034721:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.034734:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333131136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.034757:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130624:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828464.034759:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.034762:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.034764:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034767:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:3.0:1787828464.034771:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.034774:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828464.034777:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828464.034779:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.034798:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333131264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.034799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb222d80 x1874663333130624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.034810:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828464.034812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333130624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.034814:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.034816:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.034820:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10274 00000100:00080000:2.0:1787828464.034821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828464.034823:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828464.034825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828464.034827:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828464.034828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130624:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828464.034831:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828464.034835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828464.034840:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333131392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828464.034851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.034861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb221c00 x1874663333130752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.034874:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333130752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.034891:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.034895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.034898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.034912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.034929:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.034937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb222300 x1874663333130880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.034945:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333130880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.034950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.034952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.034954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333130880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828464.034958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.034967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.034973:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb220380 x1874663333131008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.034980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333131008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.034985:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.034987:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.034989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828464.035001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828464.035012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.035018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb220700 x1874663333131136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.035025:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333131136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.035042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.035044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.035046:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.035050:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828464.035061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.035067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb221880 x1874663333131264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.035076:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333131264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.035084:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.035087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.035091:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.035097:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.035117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828464.035128:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9057fb223100 x1874663333131392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.035139:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333131392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.035144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.035147:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828464.035149:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333131392:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828464.418487:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828465.442473:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828466.466695:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828467.490387:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828468.514467:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828469.154363:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154374:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828469.154377:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154389:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154401:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828469.154409:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828469.154453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333131520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828469.154484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154487:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828469.154489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154493:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154509:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828469.154514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828469.154543:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333131648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.154546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333131520:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828469.154571:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828469.154575:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154580:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154583:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828469.154586:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154591:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828469.154601:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333131776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828469.154609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828469.154613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154617:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828469.154622:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828469.154639:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828469.154660:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333131904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828469.154673:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828469.154680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828469.154682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158d880 x1874663333131520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828469.154697:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154708:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828469.154714:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333131520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828469.154718:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828469.154739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828469.154740:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828469.154749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.154754:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333131520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828469.154767:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333132032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.154779:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131648:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828469.154786:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154790:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828469.154794:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154800:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154805:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828469.154811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828469.154816:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154823:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828469.154843:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333132160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.154844:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158ce00 x1874663333131648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828469.154872:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828469.154889:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828469.154892:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154897:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00080000:0.0:1787828469.154901:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828469.154905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333131648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828469.154911:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154921:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828469.154924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.154930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828469.154935:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333132288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828469.154947:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828469.154949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131648:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828469.154953:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828469.154958:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828469.154967:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10279 00000100:00100000:2.0:1787828469.154969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131776:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828469.154973:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828469.154987:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828469.154993:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828469.155007:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333132416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828469.155017:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158c700 x1874663333131776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333131776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.155091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.155111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155121:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158d180 x1874663333131904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155131:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333131904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155139:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155141:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333131904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828469.155171:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.155188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155198:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158d500 x1874663333132032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155217:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333132032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155266:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155271:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828469.155299:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828469.155338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155373:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158ed80 x1874663333132160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155390:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333132160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155406:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155411:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.155421:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828469.155466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155481:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158fb80 x1874663333132288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155524:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333132288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155554:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155557:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155559:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.155564:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.155579:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828469.155587:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9057f158ea00 x1874663333132416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.155595:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333132416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.155604:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.155608:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828469.155611:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333132416:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828469.538511:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828470.562587:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828471.586491:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828472.612453:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828473.634643:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828474.274345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828474.274356:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828474.274412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333132544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274487:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828474.274489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274492:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274496:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274510:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828474.274530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333132672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274538:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274541:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828474.274542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274545:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274547:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274552:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274552:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333132544:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828474.274556:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828474.274562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333132800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828474.274572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274574:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274575:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274579:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828474.274594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333132928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828474.274608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274610:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274612:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274614:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828474.274625:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333133056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828474.274635:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274640:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274642:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828474.274653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828474.274654:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333133184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828474.274666:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274673:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828474.274686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333133312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828474.274692:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828474.274694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828474.274697:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158ea00 x1874663333132544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828474.274698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828474.274706:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10285 00000100:00080000:0.0:1787828474.274711:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828474.274725:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828474.274729:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828474.274731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333132544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828474.274739:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333133440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828474.274742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.274751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.274757:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333132544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828474.274764:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828474.274782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158fb80 x1874663333132672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.274799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333132672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.274805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.274807:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.274809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828474.274827:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828474.274852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158ed80 x1874663333132800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.274867:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333132800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.274873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.274875:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.274877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828474.274881:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828474.274903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158d500 x1874663333132928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.274918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333132928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.274923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.274925:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.274927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333132928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828474.274930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828474.274940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158d180 x1874663333133056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.274953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333133056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.274959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.274960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.274962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828474.274965:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828474.274975:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.274983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158c700 x1874663333133184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.274995:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333133184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.275009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.275013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.275017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828474.275023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828474.275043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.275052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158ce00 x1874663333133312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.275060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333133312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.275065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.275067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.275069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828474.275073:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828474.275084:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828474.275090:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828474/real 1787828474] req@ffff9057f158d880 x1874663333133440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828474.275097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333133440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828474.275102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828474.275103:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828474.275106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333133440:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828474.658429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828475.682500:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828476.706385:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828477.730476:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828478.754560:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828479.394459:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828479.394472:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394480:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394491:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394497:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828479.394535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333133568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828479.394562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828479.394567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394571:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394574:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394576:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394585:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828479.394598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333133696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828479.394608:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394614:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828479.394614:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333133568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828479.394619:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394628:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394632:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394635:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394640:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828479.394650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333133824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828479.394658:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394661:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828479.394662:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394665:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394668:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394671:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394675:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828479.394684:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828479.394686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333133952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828479.394695:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394698:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828479.394700:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394704:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394707:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394710:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394713:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828479.394725:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333134080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828479.394731:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78ab480 x1874663333133568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828479.394736:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394744:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828479.394748:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394755:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394758:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828479.394761:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333133568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828479.394765:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394774:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828479.394780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828479.394786:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333134208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828479.394787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.394797:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333133568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828479.394807:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394811:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828479.394813:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394816:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394819:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394821:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394824:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828479.394831:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828479.394835:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333134336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828479.394848:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828479.394851:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828479.394853:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828479.394856:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10290 00000100:00080000:1.0:1787828479.394860:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828479.394863:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828479.394867:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828479.394874:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828479.394890:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333134464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828479.394895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78aad80 x1874663333133696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.394926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333133696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.394934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.394938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.394942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828479.394948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828479.394968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.394976:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78a8000 x1874663333133824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.394985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333133824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.394991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.394994:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.394996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828479.395001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828479.395021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.395035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78abb80 x1874663333133952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.395051:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333133952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.395061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.395066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.395070:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333133952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828479.395076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828479.395093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.395112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78ab800 x1874663333134080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.395129:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333134080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.395135:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.395138:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.395140:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828479.395144:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828479.395157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.395165:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78a9880 x1874663333134208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.395174:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333134208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.395180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.395182:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.395184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828479.395200:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828479.395212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.395244:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78a8380 x1874663333134336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.395255:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333134336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.395267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.395269:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.395272:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828479.395276:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828479.395291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828479.395300:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828479/real 1787828479] req@ffff9057c78a9f80 x1874663333134464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828479.395308:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333134464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828479.395315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828479.395317:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828479.395319:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333134464:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828479.778404:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828480.802672:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828481.826469:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828482.850624:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828483.874504:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828484.514419:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514431:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828484.514437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514449:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514462:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514471:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828484.514514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333134592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828484.514564:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514570:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514574:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514578:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828484.514626:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333134720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514635:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828484.514641:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514644:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514650:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514655:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828484.514668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333134592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828484.514688:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333134848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514704:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514709:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828484.514712:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514718:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514728:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514733:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828484.514764:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333134976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514776:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514780:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828484.514783:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828484.514791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828484.514793:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514801:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514806:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514811:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828484.514823:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333135104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514833:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514837:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828484.514840:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514844:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514847:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514851:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514855:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828484.514865:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333135232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514890:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514893:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828484.514894:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514897:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514899:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514901:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514905:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828484.514912:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333135360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828484.514919:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828484.514921:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828484.514923:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828484.514925:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10295 00000100:00080000:0.0:1787828484.514927:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828484.514929:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828484.514932:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828484.514939:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333135488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828484.516211:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158e680 x1874663333134592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516241:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333134592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516276:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516282:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333134592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828484.516293:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828484.516327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516344:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158f800 x1874663333134720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333134720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516374:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516378:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828484.516454:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828484.516492:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516507:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158c000 x1874663333134848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516522:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333134848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516537:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516540:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828484.516547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828484.516567:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516578:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158dc00 x1874663333134976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516593:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333134976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516602:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516605:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516609:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333134976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828484.516615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828484.516633:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158e300 x1874663333135104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333135104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516667:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516690:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828484.516694:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828484.516709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516717:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158f100 x1874663333135232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516727:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333135232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516740:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516742:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516745:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828484.516749:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828484.516784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516796:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158ca80 x1874663333135360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516809:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333135360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828484.516832:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828484.516851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828484.516861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828484/real 1787828484] req@ffff9057f158df80 x1874663333135488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828484.516889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333135488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828484.516895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828484.516897:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828484.516900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333135488:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828484.898535:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828485.922503:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828486.946506:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828487.970595:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828488.994557:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828489.634479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828489.634495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634506:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634518:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634526:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828489.634569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333135616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828489.634608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634613:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634619:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828489.634641:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333135744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634660:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634664:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828489.634665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634681:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828489.634690:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333135616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828489.634695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333135872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634714:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828489.634720:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634723:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634727:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828489.634745:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333136000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634755:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828489.634761:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634765:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634780:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634784:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634800:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828489.634808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828489.634815:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333136128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634826:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634830:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828489.634832:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634836:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634842:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634846:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634851:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828489.634862:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333136256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634871:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634878:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828489.634880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb223100 x1874663333135616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828489.634880:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828489.634891:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634897:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634904:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828489.634908:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828489.634917:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333136384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828489.634953:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828489.634957:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828489.634959:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828489.634966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333135616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828489.634970:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10300 00000100:00080000:3.0:1787828489.634981:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828489.634994:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828489.635004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828489.635006:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828489.635021:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828489.635034:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333136512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828489.635035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333135616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828489.635060:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828489.635106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635129:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb221880 x1874663333135744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635152:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333135744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635173:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635179:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828489.635213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828489.635313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635337:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb220700 x1874663333135872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635396:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333135872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635420:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635428:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333135872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828489.635441:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828489.635494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb220380 x1874663333136000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635530:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333136000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635548:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635553:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828489.635562:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828489.635593:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635615:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb222300 x1874663333136128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635645:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333136128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635669:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828489.635678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828489.635709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635726:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb221c00 x1874663333136256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635745:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333136256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828489.635798:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828489.635836:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.635859:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb222d80 x1874663333136384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.635884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333136384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.635911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.635917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.635924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828489.635935:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828489.635972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828489.636011:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff9057fb222a00 x1874663333136512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828489.636032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333136512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828489.636048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828489.636053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828489.636060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333136512:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828490.018494:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828491.042522:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828492.066432:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828493.090589:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828494.114412:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828494.754390:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754397:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828494.754401:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754409:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754429:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754435:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828494.754468:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333136640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828494.754507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754512:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754515:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828494.754530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333136768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754538:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754541:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.754542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754546:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754549:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754551:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754554:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828494.754580:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333136896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828494.754584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333136640:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828494.754605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828494.754612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754626:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754637:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828494.754654:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333137024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.754668:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754671:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754677:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754680:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754685:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828494.754697:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828494.754701:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333137152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828494.754719:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754723:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754727:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754731:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754736:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828494.754750:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333137280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754769:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.754772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754775:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00080000:0.0:1787828494.754780:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754783:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754789:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828494.754798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158df80 x1874663333136640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828494.754802:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333137408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.754874:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.754878:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.754879:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.754884:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10305 00000100:00100000:2.0:1787828494.754884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333136640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828494.754890:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.754909:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828494.754918:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828494.754941:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333137536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828494.754943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.754959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.754975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333136640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828494.754988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828494.755036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158ca80 x1874663333136768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755069:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333136768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755107:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828494.755146:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828494.755170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755182:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158f100 x1874663333136896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755194:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333136896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755206:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755209:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333136896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828494.755214:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828494.755230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755239:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158e300 x1874663333137024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755250:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333137024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755260:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755262:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828494.755267:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828494.755282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755364:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158dc00 x1874663333137152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755377:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333137152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755387:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755390:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828494.755395:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828494.755434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755445:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158c000 x1874663333137280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755455:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333137280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755472:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755475:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828494.755480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828494.755512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755519:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158f800 x1874663333137408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755528:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333137408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755537:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755539:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828494.755543:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828494.755574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828494.755583:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff9057f158e680 x1874663333137536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828494.755592:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333137536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828494.755599:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828494.755602:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828494.755606:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333137536:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828495.138601:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828496.162453:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828497.186433:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828498.210457:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828499.234486:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828499.874346:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874354:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828499.874357:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874375:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828499.874382:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828499.874415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333137664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874444:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828499.874445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874451:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828499.874453:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828499.874490:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333137792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874501:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828499.874502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874506:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874508:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828499.874511:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828499.874517:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333137664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828499.874543:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333137920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874553:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874556:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828499.874557:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874560:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874563:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828499.874566:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874570:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828499.874580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333138048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874589:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828499.874591:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874593:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874598:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828499.874598:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828499.874606:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828499.874619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333138176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874627:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874630:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828499.874632:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874635:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874638:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828499.874642:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158d880 x1874663333137664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828499.874645:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874653:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828499.874669:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333138304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828499.874672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333137664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828499.874693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828499.874697:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828499.874698:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828499.874702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.874705:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828499.874709:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00100000:2.0:1787828499.874713:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333137664:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828499.874715:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828499.874724:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828499.874728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137792:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828499.874729:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828499.874746:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333138432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828499.874754:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828499.874755:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828499.874761:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828499.874763:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828499.874768:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10310 00000100:00080000:0.0:1787828499.874772:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828499.874772:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158ce00 x1874663333137792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828499.874779:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828499.874789:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828499.874795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333137792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828499.874801:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333138560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828499.874811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.874819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.874825:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828499.874869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828499.874891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.874901:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158c700 x1874663333137920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.874909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333137920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.874914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.874916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.874918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333137920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828499.874922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828499.874935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.874940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158d180 x1874663333138048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.874947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333138048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.874952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.874954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.874956:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828499.874959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828499.874969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.874975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158d500 x1874663333138176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.874987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333138176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.874992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.874993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.874995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828499.874998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828499.875008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.875014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158ed80 x1874663333138304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.875021:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333138304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.875026:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.875027:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.875029:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828499.875032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828499.875042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.875047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158fb80 x1874663333138432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.875053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333138432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.875062:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.875064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.875066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828499.875070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828499.875079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828499.875085:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff9057f158ea00 x1874663333138560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828499.875091:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333138560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828499.875096:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828499.875098:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828499.875100:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333138560:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828500.258457:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828501.282547:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828502.306588:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828503.330411:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828504.354457:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828504.994457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994469:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828504.994474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994485:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994497:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994505:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828504.994549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333138688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.994573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828504.994578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994581:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994584:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994587:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828504.994610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828504.994628:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333138816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.994642:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828504.994650:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994655:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994668:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828504.994677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333138688:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828504.994679:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828504.994708:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333138944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.994720:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994724:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828504.994725:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994729:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994732:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994735:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828504.994739:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828504.994751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333139072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.994761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994764:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828504.994765:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994769:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994772:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994776:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828504.994782:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828504.994790:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333139200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828504.994820:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828504.994836:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994842:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828504.994844:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994853:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994859:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994875:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828504.994882:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828504.994901:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333139328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.994913:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828504.994918:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828504.994918:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78aaa00 x1874663333138688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.994928:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.994946:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.994962:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.994972:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828504.994975:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333138688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828504.994988:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828504.994997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333138688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828504.995021:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333139456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828504.995073:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828504.995075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138816:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828504.995095:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828504.995101:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828504.995108:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10315 00000100:00080000:1.0:1787828504.995118:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828504.995124:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828504.995132:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828504.995142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828504.995150:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333139584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828504.995181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78a8a80 x1874663333138816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995210:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333138816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995299:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995304:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995309:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828504.995348:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828504.995380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995395:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78ab100 x1874663333138944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995409:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333138944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995423:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995427:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333138944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828504.995434:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828504.995455:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995466:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78a9500 x1874663333139072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995480:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333139072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995493:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995496:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828504.995503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828504.995520:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995552:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78a8700 x1874663333139200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995564:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333139200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995576:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995579:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828504.995584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828504.995618:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78a9c00 x1874663333139328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995641:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333139328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995658:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995662:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995666:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828504.995672:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828504.995692:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995702:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78aa300 x1874663333139456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333139456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995730:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995737:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828504.995743:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828504.995760:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828504.995770:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff9057c78a9180 x1874663333139584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.995799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333139584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828504.995807:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828504.995810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828504.995814:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333139584:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828505.378751:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828506.402342:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828507.426505:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828508.450576:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828509.474548:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828510.114354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.114366:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828510.114381:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114394:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114421:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.114432:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828510.114478:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333139712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.114513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.114517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828510.114520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114529:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.114533:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.114547:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828510.114564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333139840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.114581:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.114590:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828510.114590:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333139712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828510.114599:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114611:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114615:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.114619:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.114626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828510.114642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333139968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.114657:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.114661:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828510.114663:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114678:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114685:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.114692:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.114703:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828510.114728:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333140096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.114749:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828510.114763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139840:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828510.114778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828510.114785:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114799:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114810:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828510.114817:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139968:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828510.114827:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.114843:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828510.114865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828510.114881:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333140224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.114909:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.114916:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828510.114920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140224:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787828510.114924:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.114942:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.114953:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.114963:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.114973:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828510.115012:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333140352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.115046:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.115060:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828510.115063:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.115068:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.115075:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.115082:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.115090:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828510.115106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828510.115114:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333140480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828510.115132:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828510.115136:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828510.115140:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828510.115145:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10320 00000100:00080000:1.0:1787828510.115151:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828510.115157:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828510.115165:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828510.115170:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828510.115189:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333140608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828510.115282:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140608:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828510.498544:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828511.522483:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828512.546437:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828513.186545:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828513.186596:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828513.186616:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186724:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186736:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186758:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186769:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186778:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186788:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828513.186801:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828513.186855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78a8a80 x1874663333140608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.186888:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333140608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.186904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.186910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.186915:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828513.186925:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78ab100 x1874663333140480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.186941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333140480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.186951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.186954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.186958:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828513.186967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78a9500 x1874663333140352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.186980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333140352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.186988:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.186992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.186996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333140352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828513.187004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78a8700 x1874663333140224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.187016:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333140224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.187024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.187029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.187033:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828513.187040:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78a9180 x1874663333139840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.187053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333139840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.187061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.187065:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.187069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828513.187076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78aa300 x1874663333139968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.187088:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333139968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.187097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.187100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.187104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333139968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828513.187112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78a9c00 x1874663333140096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.187123:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333140096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.187132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.187135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.187139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828513.187146:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828513] req@ffff9057c78aa680 x1874663333139712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828513.187159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333139712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828513.187168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828513.187172:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828513.187174:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333139712:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1787828513.570484:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828514.594421:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828515.234476:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828515.234490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828515.234496:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.234511:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.234528:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828515.234539:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828515.234594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333140736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828515.234640:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828515.234645:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828515.234647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.234652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.234656:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828515.234662:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.234679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828515.234705:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333140864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828515.234726:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828515.234735:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828515.234737:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333140736:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828515.234739:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.234753:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.234763:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828515.234770:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.234780:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828515.234803:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333140992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828515.234821:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828515.234827:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828515.234831:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.234837:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.234856:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828515.234864:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.234874:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828515.234875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828515.234900:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333141120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828515.234919:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828515.234925:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828515.234929:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.234947:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.234955:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828515.234959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb222680 x1874663333140736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828515.234966:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.234981:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828515.235012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333140736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828515.235013:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333141248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828515.235063:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828515.235070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828515.235071:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828515.235079:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828515.235088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828515.235095:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00100000:2.0:1787828515.235106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333140736:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828515.235123:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828515.235133:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.235143:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828515.235166:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333141376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828515.235172:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828515.235290:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828515.235300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828515.235304:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828515.235311:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828515.235322:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.235331:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828515.235335:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb223800 x1874663333140864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828515.235344:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828515.235361:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828515.235375:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333140864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828515.235390:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333141504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828515.235429:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828515.235434:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828515.235444:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828515.235446:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828515.235455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828515.235460:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828515.235467:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10325 00000100:00080000:3.0:1787828515.235480:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828515.235487:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140992:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828515.235494:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828515.235536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828515.235559:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb221500 x1874663333140992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.235600:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333140992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.235617:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.235625:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.235631:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333140992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828515.235661:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828515.235695:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828515.235714:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb221180 x1874663333141120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.235737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333141120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.235752:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.235759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.235765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828515.235777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141248:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787828515.235807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828515.235821:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828515.235839:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333141632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828515.235853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb220000 x1874663333141248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.235885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333141248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.235902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.235909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.235917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828515.235956:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828515.235995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828515.236027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb223b80 x1874663333141376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.236049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333141376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.236064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.236069:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.236075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828515.236086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828515.236119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828515.236139:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb220a80 x1874663333141504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.236200:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333141504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.236209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.236214:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.236217:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828515.236251:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828515.236276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828515.236290:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828515/real 1787828515] req@ffff9057fb220e00 x1874663333141632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828515.236306:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333141632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828515.236316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828515.236320:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828515.236325:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333141632:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828515.618464:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828516.642521:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828517.666372:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828518.690374:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828519.714504:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828520.354312:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354320:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828520.354324:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354332:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354340:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354346:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828520.354378:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333141760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354399:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354403:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828520.354405:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354407:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354410:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354412:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354419:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828520.354428:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333141888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354448:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828520.354454:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333141760:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828520.354455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354462:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354468:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354471:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828520.354480:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333142016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354487:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354489:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828520.354491:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354494:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354497:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354499:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354504:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828520.354511:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828520.354513:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333142144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354521:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354524:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828520.354525:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354528:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354531:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354534:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354538:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828520.354551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333142272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828520.354563:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78aaa00 x1874663333141760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828520.354572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828520.354577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828520.354594:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333141760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828520.354595:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354603:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828520.354609:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828520.354610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828520.354615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828520.354616:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354622:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333141760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828520.354642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333142400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354650:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828520.354652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828520.354654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354657:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828520.354663:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141888:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828520.354666:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354672:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828520.354682:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333142528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828520.354690:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828520.354694:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828520.354696:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828520.354699:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828520.354703:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10331 00000100:00080000:1.0:1787828520.354707:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828520.354711:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78a9f80 x1874663333141888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828520.354713:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828520.354720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828520.354728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333141888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828520.354732:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333142656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828520.354742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354750:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354754:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333141888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828520.354759:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828520.354773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.354779:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78a8380 x1874663333142016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.354787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333142016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.354792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354795:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828520.354800:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828520.354811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.354816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78a9880 x1874663333142144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.354823:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333142144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.354837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828520.354843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828520.354853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.354858:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78ab800 x1874663333142272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.354865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333142272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.354870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354872:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828520.354877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828520.354886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.354892:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78abb80 x1874663333142400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.354899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333142400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.354904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354907:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828520.354925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828520.354940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.354951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78a8000 x1874663333142528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.354963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333142528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.354972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.354976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.354987:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828520.354993:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828520.355008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828520.355021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828520/real 1787828520] req@ffff9057c78aad80 x1874663333142656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828520.355028:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333142656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828520.355039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828520.355041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828520.355043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333142656:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828520.738419:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828521.762481:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828522.786447:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828523.810595:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828524.834500:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828525.474338:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828525.474360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00080000:1.0:1787828525.474385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474395:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828525.474441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333142784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828525.474478:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474483:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828525.474486:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474491:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00080000:1.0:1787828525.474495:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474502:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828525.474517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828525.474543:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333142912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828525.474566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828525.474578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474587:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00080000:1.0:1787828525.474596:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474605:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828525.474616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333142784:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828525.474618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828525.474643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333143040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828525.474659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474664:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828525.474668:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474673:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00080000:1.0:1787828525.474679:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474684:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828525.474691:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828525.474704:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333143168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828525.474717:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474721:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828525.474724:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474735:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10336 00000800:00000100:2.0:1787828525.474735:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828525.474745:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474752:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828525.474759:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828525.474775:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333143296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828525.474790:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474798:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828525.474802:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828525.474807:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78ab480 x1874663333142784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828525.474817:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00100000:2.0:1787828525.474843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333142784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828525.474850:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828525.474868:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828525.474870:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828525.474883:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787828525.474889:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828525.474892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333142784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828525.474910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828525.474935:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333143424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828525.474953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828525.474954:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.474963:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828525.474966:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.474974:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00100000:2.0:1787828525.474981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78aad80 x1874663333142912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828525.474985:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.474999:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828525.475014:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828525.475022:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333142912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828525.475038:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333143552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828525.475040:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475056:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333142912:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828525.475075:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828525.475086:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828525.475089:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828525.475096:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10336 00000100:00080000:1.0:1787828525.475125:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828525.475132:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828525.475139:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828525.475149:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828525.475163:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333143680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828525.475191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475214:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78a8000 x1874663333143040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475281:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333143040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475296:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475300:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828525.475308:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828525.475333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475346:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78abb80 x1874663333143168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333143168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475373:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475377:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828525.475383:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828525.475402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475413:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78ab800 x1874663333143296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475428:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333143296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475441:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475444:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828525.475451:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828525.475469:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475480:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78a9880 x1874663333143424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475493:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333143424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475506:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475510:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828525.475516:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828525.475535:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475545:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78a8380 x1874663333143552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475558:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333143552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475575:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475579:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475584:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828525.475605:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828525.475638:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828525.475658:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828525/real 1787828525] req@ffff9057c78a9f80 x1874663333143680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828525.475683:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333143680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828525.475701:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828525.475708:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828525.475715:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333143680:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828525.858546:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828526.882483:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828527.906507:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828528.930441:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828529.954498:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828530.594331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828530.594347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594370:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594392:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828530.594457:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333143808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594492:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828530.594497:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828530.594542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333143936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594554:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828530.594557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828530.594560:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333143808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828530.594561:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594566:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594571:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594575:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828530.594584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333144064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828530.594599:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594602:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594604:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594607:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828530.594625:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333144192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828530.594635:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594637:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594644:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828530.594653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828530.594657:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333144320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594667:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828530.594669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594672:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594679:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594681:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828530.594691:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333144448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828530.594702:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828530.594713:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb220e00 x1874663333143808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828530.594715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828530.594719:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594728:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594733:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594735:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828530.594740:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333143808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828530.594750:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333144576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828530.594753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.594762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828530.594762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828530.594769:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828530.594771:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333143808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828530.594772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828530.594780:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10341 00000100:00080000:3.0:1787828530.594782:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828530.594784:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828530.594787:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828530.594795:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333144704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828530.594801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828530.594829:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.594841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb220a80 x1874663333143936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.594849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333143936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.594855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.594858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.594860:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333143936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828530.594864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828530.594875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.594881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb223b80 x1874663333144064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.594904:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333144064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.594910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.594912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.594914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828530.594930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828530.594945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.594952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb220000 x1874663333144192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.594960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333144192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.594966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.594968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.594970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828530.594974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828530.594985:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.594992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb221180 x1874663333144320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.595000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333144320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.595007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.595009:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.595011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828530.595014:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828530.595026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.595032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb221500 x1874663333144448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.595046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333144448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.595052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.595054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.595056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828530.595059:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828530.595087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.595096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb223800 x1874663333144576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.595121:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333144576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.595127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.595129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.595131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828530.595145:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828530.595157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828530.595164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff9057fb222680 x1874663333144704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828530.595171:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333144704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828530.595181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828530.595183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828530.595185:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333144704:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828530.978654:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828532.002502:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828533.026410:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828534.050571:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828535.074446:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828535.714427:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714439:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828535.714443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714456:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714471:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714480:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828535.714533:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333144832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714578:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714581:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828535.714582:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714586:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714589:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714592:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828535.714611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333144960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714618:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.714622:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714624:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714626:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714628:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714631:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828535.714637:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333145088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714643:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828535.714659:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714661:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714663:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714666:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828535.714676:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333145216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714682:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.714686:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714689:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714691:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714693:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714695:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828535.714702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333145344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714707:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828535.714711:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714714:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714716:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714722:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714725:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828535.714731:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333145472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714749:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714752:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.714754:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714757:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714759:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714761:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714764:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828535.714769:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333145600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.714775:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.714777:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.714778:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.714782:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10346 00000100:00080000:0.0:1787828535.714784:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.714786:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828535.714790:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828535.714799:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333145728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828535.714809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333144832:192.168.203.156@tcp:250:kworker.0 00000800:00000100:2.0:1787828535.714928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.714999:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158ea00 x1874663333144832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333144832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715099:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715105:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333144832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828535.715115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828535.715166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158fb80 x1874663333144960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715199:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333144960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715220:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715225:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715229:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333144960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828535.715237:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828535.715343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715377:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158ed80 x1874663333145088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715393:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333145088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715407:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715411:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828535.715427:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828535.715453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715466:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158d500 x1874663333145216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715480:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333145216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715493:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715496:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828535.715503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828535.715521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715533:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158d180 x1874663333145344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715547:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333145344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715577:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715580:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828535.715587:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828535.715608:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715619:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158c700 x1874663333145472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715632:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333145472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715640:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715644:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715650:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828535.715660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828535.715693:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158ce00 x1874663333145600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715743:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333145600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715766:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715770:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715774:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828535.715849:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828535.715880:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828535.715911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff9057f158d880 x1874663333145728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828535.715924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333145728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828535.715934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828535.715938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828535.715941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333145728:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828536.098482:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828537.122441:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828538.146488:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828539.170384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828540.194428:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828540.834331:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834343:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828540.834349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834361:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834374:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834384:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828540.834435:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333145856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.834472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834477:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828540.834480:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834485:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834490:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834494:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828540.834522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333145984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.834538:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834543:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.834546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828540.834555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333145856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828540.834557:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834568:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834574:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834581:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828540.834597:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333146112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.834620:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828540.834627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834631:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834637:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834641:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828540.834654:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828540.834669:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333146240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.834698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834708:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.834711:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834716:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834721:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834725:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834731:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828540.834744:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158e680 x1874663333145856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828540.834752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333146368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.834814:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834819:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828540.834821:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834827:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.834835:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828540.834838:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333145856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828540.834846:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834860:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828540.834876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828540.834892:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333146496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828540.834893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.834912:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333145856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828540.834937:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.834945:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.834950:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.834958:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00100000:2.0:1787828540.834963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145984:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828540.834967:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.834981:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.834990:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828540.835036:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333146624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.835052:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.835056:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.835058:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.835063:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10351 00000100:00080000:0.0:1787828540.835068:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.835072:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828540.835078:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828540.835089:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333146752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828540.835197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835277:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158f800 x1874663333145984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835297:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333145984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835313:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835318:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333145984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828540.835327:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828540.835365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835387:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158c000 x1874663333146112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835412:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333146112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835436:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835444:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828540.835456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828540.835487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835502:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158dc00 x1874663333146240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835517:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333146240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835531:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835535:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828540.835542:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828540.835561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835572:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158e300 x1874663333146368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835586:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333146368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835600:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835603:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333146368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828540.835610:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828540.835628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835640:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158f100 x1874663333146496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835662:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333146496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835672:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835675:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835679:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828540.835704:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828540.835723:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835735:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158ca80 x1874663333146624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835748:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333146624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828540.835771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828540.835789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828540.835799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff9057f158df80 x1874663333146752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828540.835812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333146752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828540.835830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828540.835834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828540.835837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333146752:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828541.218443:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828542.242609:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828543.266350:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828544.290549:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828545.314492:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828545.954389:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.954406:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828545.954413:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.954430:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.954446:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.954458:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828545.954512:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333146880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.954563:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.954571:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828545.954575:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.954583:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.954591:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.954600:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.954616:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828545.954662:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333146880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828545.954663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333147008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.954690:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.954695:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.954697:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.954703:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.954708:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.954714:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.954722:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828545.954736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333147136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.954748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.954755:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828545.954757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828545.954760:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.954768:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.954774:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.954779:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.954786:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828545.954801:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333147264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.954814:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828545.954820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158df80 x1874663333146880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828545.954822:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.954828:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.954842:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.954851:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.954882:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.954890:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828545.954896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333146880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828545.954911:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333147392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828545.954919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.954935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828545.954936:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828545.954949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333146880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828545.954950:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828545.954957:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.955005:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.955012:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.955019:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.955026:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828545.955041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828545.955048:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333147520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.955075:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.955081:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.955084:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.955089:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.955095:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828545.955102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828545.955107:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.955116:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828545.955144:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333147648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828545.955152:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158ca80 x1874663333147008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828545.955168:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.955189:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.955197:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828545.955208:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333147008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828545.955210:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10356 00000100:00080000:0.0:1787828545.955280:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.955296:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828545.955305:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828545.955328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955344:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828545.955347:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333147776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828545.955357:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.955381:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828545.955425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955455:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158f100 x1874663333147136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955481:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333147136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955498:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955504:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955512:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.955525:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828545.955563:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158e300 x1874663333147264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955595:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333147264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955613:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828545.955620:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828545.955648:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955659:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158dc00 x1874663333147392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955674:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333147392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955690:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333147392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828545.955713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828545.955733:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955745:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158c000 x1874663333147520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955758:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333147520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955771:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955774:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.955781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828545.955800:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158f800 x1874663333147648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333147648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955853:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955857:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.955863:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828545.955882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828545.955892:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff9057f158e680 x1874663333147776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.955905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333147776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.955914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.955918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828545.955922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333147776:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828546.338471:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828547.362502:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828548.386476:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828549.410363:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828550.434415:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828551.074336:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.074348:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828551.074353:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.074365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.074378:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.074390:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828551.074447:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333147904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.074506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.074510:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828551.074513:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.074525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.074531:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.074537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.074560:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.074580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333148032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.074595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.074600:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828551.074604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.074610:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.074621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828551.074623:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333147904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828551.074636:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.074649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.074677:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333148160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.074710:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.074715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828551.074717:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.074739:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.074750:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828551.074751:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828551.074764:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.074775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828551.074801:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333148288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.074820:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.074830:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828551.074833:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158d880 x1874663333147904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.074836:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.074852:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.074873:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.074885:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.074894:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828551.074906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333147904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.074915:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333148416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.074994:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828551.074997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.075000:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828551.075007:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.075020:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:2.0:1787828551.075020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828551.075029:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.075040:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333147904:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828551.075046:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.075076:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333148544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828551.075076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148032:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828551.075106:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.075109:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828551.075112:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.075117:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.075120:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.075125:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.075146:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828551.075150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828551.075166:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333148672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.075181:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.075189:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828551.075191:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158ce00 x1874663333148032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.075193:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828551.075204:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10361 00000100:00080000:0.0:1787828551.075214:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.075261:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828551.075267:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828551.075283:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333148800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828551.075301:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333148032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.075350:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.075360:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828551.075403:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828551.075447:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075477:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158c700 x1874663333148160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.075500:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333148160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075515:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.075521:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.075528:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828551.075550:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828551.075586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075608:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158d180 x1874663333148288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.075632:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333148288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.075653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.075661:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828551.075672:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828551.075720:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075743:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158d500 x1874663333148416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.075765:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333148416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075781:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.075787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.075793:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333148416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828551.075805:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828551.075841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158ed80 x1874663333148544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.075885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333148544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.075899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.075903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828551.075910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828551.075959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.075974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158fb80 x1874663333148672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.075988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333148672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.075997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.076001:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.076004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828551.076011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828551.076032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828551.076042:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9057f158ea00 x1874663333148800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828551.076054:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333148800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828551.076071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828551.076074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828551.076078:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333148800:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828551.458432:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828552.482511:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828553.506412:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828554.530544:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828555.554490:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828556.194343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194351:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828556.194354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828556.194378:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828556.194410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333148928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828556.194433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194435:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828556.194437:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828556.194445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194452:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828556.194461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333149056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828556.194470:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828556.194471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333148928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828556.194474:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828556.194479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194482:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194484:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828556.194487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194491:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828556.194502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333149184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828556.194513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828556.194519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194528:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828556.194531:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828556.194536:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194559:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828556.194573:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333149312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828556.194583:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78aaa00 x1874663333148928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828556.194584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828556.194596:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194604:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194607:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828556.194611:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333148928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828556.194614:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828556.194632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333149440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828556.194640:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828556.194641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828556.194644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828556.194647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828556.194651:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828556.194654:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194657:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828556.194661:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333148928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828556.194662:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194670:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828556.194678:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333149568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828556.194684:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828556.194690:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828556.194691:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149056:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828556.194696:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00080000:1.0:1787828556.194699:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828556.194702:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828556.194714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333149696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828556.194721:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828556.194724:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828556.194728:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828556.194730:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828556.194734:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10366 00000100:00100000:2.0:1787828556.194737:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78aa680 x1874663333149056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828556.194738:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828556.194745:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828556.194753:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828556.194757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333149056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.194765:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.194768:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828556.194772:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333149824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828556.194773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828556.194780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828556.194802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.194813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78a9c00 x1874663333149184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.194821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333149184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.194826:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.194828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.194830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828556.194833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828556.194844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.194850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78aa300 x1874663333149312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.194857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333149312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.194861:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.194863:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.194865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828556.194868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828556.194878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.194884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78a9180 x1874663333149440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.194907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333149440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.194913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.194915:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.194917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333149440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828556.194921:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828556.194956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.194967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78a8700 x1874663333149568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.194979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333149568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.194987:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.194990:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.194993:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828556.195011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828556.195039:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.195045:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78a9500 x1874663333149696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.195052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333149696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.195058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.195060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.195062:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828556.195065:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828556.195078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828556.195088:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9057c78ab100 x1874663333149824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828556.195099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333149824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828556.195108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828556.195111:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828556.195115:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333149824:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828556.578407:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828557.602426:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828558.626431:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828559.650544:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828560.674475:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828561.314287:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314295:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828561.314298:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314307:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314318:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314326:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828561.314367:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333149952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314392:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314395:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828561.314397:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314400:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314403:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314407:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314414:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828561.314425:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333150080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828561.314438:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314442:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314447:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828561.314459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333150208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314475:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828561.314480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314487:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00100000:2.0:1787828561.314489:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333149952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828561.314490:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314500:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314506:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828561.314520:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333150336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828561.314531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314539:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828561.314553:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333150464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828561.314564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314567:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314569:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314572:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828561.314581:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828561.314588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333150592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828561.314602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314606:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314611:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828561.314614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828561.314620:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333150720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828561.314626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828561.314628:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828561.314631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828561.314634:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10372 00000100:00080000:1.0:1787828561.314637:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828561.314644:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828561.314644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78a8a80 x1874663333149952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828561.314650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828561.314658:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333150848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828561.314749:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333149952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.314763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.314767:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.314771:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333149952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828561.314777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828561.314798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.314808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78ab100 x1874663333150080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.314816:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333150080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.314821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.314824:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.314826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828561.314872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828561.314888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.314902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78a9500 x1874663333150208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.314911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333150208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.314916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.314919:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.314921:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828561.314925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828561.314939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.314945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78a8700 x1874663333150336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.314953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333150336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.314958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.314960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.314961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828561.314965:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828561.314976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.314982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78a9180 x1874663333150464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.314989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333150464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.314994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.314996:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.314998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333150464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828561.315001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828561.315016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.315026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78aa300 x1874663333150592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.315046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333150592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.315056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.315059:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.315063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828561.315069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828561.315086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.315094:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78a9c00 x1874663333150720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.315102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333150720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.315108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.315110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.315112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828561.315115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828561.315126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828561.315132:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828561/real 1787828561] req@ffff9057c78aa680 x1874663333150848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828561.315139:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333150848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828561.315149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828561.315151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828561.315153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333150848:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828561.698460:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828562.722387:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828563.746453:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828564.770702:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828565.794547:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828566.434308:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434316:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828566.434319:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434327:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434336:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828566.434373:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333150976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434397:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434400:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828566.434401:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434404:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434406:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434408:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434415:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828566.434424:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333151104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828566.434436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434438:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434440:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434442:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828566.434451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333151232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434459:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434462:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828566.434463:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434466:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434467:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434471:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828566.434481:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333150976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828566.434486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333151360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828566.434507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434509:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434511:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434514:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828566.434528:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333151488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828566.434536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434540:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434543:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434546:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828566.434556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333151616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434562:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828566.434564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828566.434565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434568:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434570:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434573:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828566.434583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333151744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828566.434593:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828566.434597:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828566.434598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828566.434603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828566.434607:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10377 00000100:00080000:1.0:1787828566.434611:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828566.434613:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828566.434616:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828566.434622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333151872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828566.434667:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78aaa00 x1874663333150976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.434693:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333150976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.434707:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.434713:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.434718:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333150976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828566.434728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828566.434757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.434773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78a9f80 x1874663333151104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.434786:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333151104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.434796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.434801:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.434805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828566.434856:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828566.434881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.434896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78a8380 x1874663333151232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.434910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333151232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.434918:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.434922:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.434926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828566.434933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828566.434955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.434967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78a9880 x1874663333151360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.434980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333151360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.434989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.434993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.434997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828566.435004:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828566.435025:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.435038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78ab800 x1874663333151488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.435050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333151488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.435059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.435063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.435067:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333151488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828566.435074:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828566.435094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.435107:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78abb80 x1874663333151616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.435120:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333151616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.435129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.435133:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.435136:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828566.435143:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828566.435165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.435177:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78a8000 x1874663333151744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.435190:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333151744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.435199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.435203:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.435206:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828566.435213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828566.435274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828566.435288:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828566/real 1787828566] req@ffff9057c78aad80 x1874663333151872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828566.435301:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333151872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828566.435310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828566.435314:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828566.435329:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333151872:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828566.818569:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828567.842392:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828568.866551:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828569.890459:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828570.914499:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828571.554338:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554349:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828571.554353:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554363:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554373:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828571.554414:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333152000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554438:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554440:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828571.554442:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554446:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554448:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554452:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828571.554469:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333152128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554479:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828571.554479:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333152000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828571.554487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828571.554490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554493:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554495:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554501:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828571.554508:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333152256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828571.554519:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554529:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554532:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554534:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554538:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828571.554541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828571.554550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333152384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554560:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828571.554561:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554564:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554567:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554569:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554573:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828571.554577:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158ea00 x1874663333152000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828571.554583:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333152512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554589:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828571.554599:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554604:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554607:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554609:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554613:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333152000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828571.554614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828571.554627:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828571.554631:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333152640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828571.554634:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554644:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333152000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828571.554647:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554651:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828571.554652:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554656:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10382 00000100:00080000:0.0:1787828571.554658:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554661:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554664:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828571.554668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828571.554673:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333152768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828571.554682:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828571.554685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828571.554686:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828571.554690:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10382 00000800:00000100:2.0:1787828571.554691:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828571.554693:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828571.554697:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828571.554701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828571.554707:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158fb80 x1874663333152128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828571.554711:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333152896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828571.554735:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333152128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554749:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554755:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.554760:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828571.554773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158ed80 x1874663333152256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.554805:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333152256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.554820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828571.554835:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554842:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158d500 x1874663333152384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.554849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333152384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554856:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554857:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828571.554861:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828571.554871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158d180 x1874663333152512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.554884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333152512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554889:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333152512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828571.554895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828571.554905:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158c700 x1874663333152640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.554923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333152640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.554944:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828571.554954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158ce00 x1874663333152768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.554966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333152768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.554971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.554973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.554975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.554978:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828571.554988:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828571.554993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828571/real 1787828571] req@ffff9057f158d880 x1874663333152896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.555000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333152896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.555009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.555011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828571.555013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333152896:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828571.938484:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828572.962555:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828573.986504:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828575.010514:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828576.034406:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828576.674354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828576.674370:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674380:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674391:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674411:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828576.674448:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333153024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.674480:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674483:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828576.674485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674491:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674495:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674500:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674510:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828576.674524:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333153152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.674537:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674540:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.674542:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674551:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00100000:2.0:1787828576.674554:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333153024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828576.674556:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674570:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828576.674600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333153280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.674616:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674622:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828576.674626:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674633:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674640:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674648:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828576.674678:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333153408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.674692:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674696:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.674698:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674704:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674710:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828576.674713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153152:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828576.674722:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674730:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828576.674773:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333153536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828576.674780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153280:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828576.674804:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674808:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828576.674810:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674824:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674833:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674839:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674846:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828576.674863:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333153664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828576.674865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153408:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787828576.674894:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.674900:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.674904:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.674913:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.674921:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.674930:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.674944:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828576.674946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828576.674976:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333153792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828576.674987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153664:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828576.675027:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.675034:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.675038:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828576.675044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153792:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828576.675053:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10387 00000100:00080000:1.0:1787828576.675061:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.675069:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828576.675079:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828576.675102:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333153920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828576.675186:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153920:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828577.058400:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828578.082434:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828579.106472:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828579.746431:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828579.746468:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828579.746481:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746512:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746519:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746525:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746533:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746542:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746550:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828579.746559:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828579.746585:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78a9f80 x1874663333153920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746606:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333153920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746624:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746629:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828579.746637:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78a8380 x1874663333153792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746650:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333153792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746658:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746662:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746666:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.746672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78ab800 x1874663333153536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746684:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333153536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746692:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746696:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828579.746706:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78a9880 x1874663333153664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746720:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333153664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746745:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746754:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333153664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.746761:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78aad80 x1874663333153152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333153152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746782:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746785:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746789:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.746796:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78a8000 x1874663333153280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333153280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746822:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.746829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78abb80 x1874663333153408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333153408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746848:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746851:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746855:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333153408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828579.746861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828579] req@ffff9057c78ab480 x1874663333153024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.746873:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333153024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.746880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.746884:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828579.746887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333153024:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1787828580.130338:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828581.154570:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828581.794316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828581.794352:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794367:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794383:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794413:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828581.794484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333154048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794516:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794520:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828581.794522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794526:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794534:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794538:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828581.794549:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828581.794561:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333154176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794574:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828581.794575:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794578:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794580:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794586:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828581.794590:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333154048:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828581.794592:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828581.794604:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333154304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794613:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794616:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828581.794622:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794626:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794628:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794632:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828581.794641:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828581.794651:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333154432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794660:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794667:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828581.794669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794672:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794677:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794682:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828581.794688:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828581.794691:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828581.794707:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333154560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794741:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794746:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828581.794749:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794756:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794762:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828581.794768:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828581.794775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828581.794780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78aaa00 x1874663333154048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828581.794794:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333154688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794812:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794818:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828581.794821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333154048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828581.794824:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794837:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794847:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828581.794852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828581.794856:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828581.794864:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787828581.794877:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828581.794877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333154048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828581.794895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828581.794901:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333154816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828581.794917:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828581.794922:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828581.794925:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828581.794931:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10392 00000100:00080000:1.0:1787828581.794938:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828581.794938:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828581.794949:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828581.794957:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828581.794966:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78aa680 x1874663333154176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828581.794976:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333154944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828581.795037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333154176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828581.795103:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828581.795134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795149:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78a9c00 x1874663333154304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795164:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333154304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795178:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795182:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828581.795204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828581.795280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795297:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78aa300 x1874663333154432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795312:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333154432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795336:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795340:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828581.795362:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828581.795390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795405:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78a9180 x1874663333154560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795421:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333154560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795436:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795440:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333154560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828581.795447:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828581.795474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795489:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78a8700 x1874663333154688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795505:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333154688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795520:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795525:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828581.795534:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828581.795560:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795579:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78a9500 x1874663333154816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795591:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333154816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795599:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795601:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795605:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828581.795611:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828581.795633:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828581.795643:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff9057c78ab100 x1874663333154944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.795653:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333154944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.795660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.795663:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828581.795667:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333154944:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828582.178434:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828583.202392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828584.226409:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828585.250401:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828586.274415:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828586.914358:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828586.914368:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914376:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914385:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914390:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828586.914423:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333155072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.914447:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914451:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828586.914453:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914457:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914461:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914465:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828586.914475:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828586.914487:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333155200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.914497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828586.914502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828586.914503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333155072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828586.914506:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914512:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828586.914517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828586.914527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333155328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.914535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828586.914539:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914543:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914554:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914557:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828586.914573:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828586.914574:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828586.914583:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333155456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.914591:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914594:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828586.914596:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914599:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914602:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914605:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914610:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb222a00 x1874663333155072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828586.914610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828586.914619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333155584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828586.914627:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333155072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828586.914627:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914631:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828586.914633:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914636:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:2.0:1787828586.914638:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828586.914639:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828586.914642:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828586.914642:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914646:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333155072:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828586.914647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828586.914655:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333155712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.914663:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828586.914668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155200:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828586.914668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914672:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00080000:3.0:1787828586.914689:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914694:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828586.914699:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828586.914717:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828586.914721:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333155840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828586.914729:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb222d80 x1874663333155200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828586.914731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.914735:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828586.914737:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.914741:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10397 00000100:00100000:2.0:1787828586.914742:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333155200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828586.914744:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.914748:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828586.914750:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828586.914752:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828586.914753:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914757:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828586.914761:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333155968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828586.914763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828586.914781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914792:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb221c00 x1874663333155328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.914812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333155328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.914818:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.914820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914822:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828586.914826:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828586.914841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914848:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb222300 x1874663333155456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.914855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333155456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.914860:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.914862:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828586.914867:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828586.914877:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb220380 x1874663333155584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.914889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333155584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.914898:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.914900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333155584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828586.914905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828586.914914:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914919:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb220700 x1874663333155712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.914926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333155712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.914931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.914933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828586.914947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828586.914957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb221880 x1874663333155840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.914969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333155840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.914974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.914976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.914978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828586.914982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828586.914991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828586.914996:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff9057fb223100 x1874663333155968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.915003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333155968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.915009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.915010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828586.915012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333155968:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828587.298450:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828588.322445:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828589.346369:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828590.370423:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828591.394413:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828592.034266:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034274:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828592.034277:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034285:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034294:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034300:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828592.034334:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333156096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034363:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.034369:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034373:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034376:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034380:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034390:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828592.034404:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333156224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034415:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034419:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.034422:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034426:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034430:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034435:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034439:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828592.034448:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333156352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034458:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.034460:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034463:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034465:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034467:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034472:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828592.034485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333156096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828592.034485:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333156480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.034513:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034515:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034517:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034520:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828592.034531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333156608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034537:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.034540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034543:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034545:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034566:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828592.034584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333156736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828592.034610:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828592.034617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.034621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034630:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034637:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034685:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828592.034705:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb223100 x1874663333156096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828592.034705:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333156864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.034774:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.034778:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.034780:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.034782:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10402 00000100:00080000:3.0:1787828592.034787:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.034792:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828592.034795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828592.034798:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333156096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828592.034807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333156992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828592.034818:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.034826:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.034832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333156096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828592.034840:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828592.034870:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.034885:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb221880 x1874663333156224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.034900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333156224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.034906:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.034908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.034910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828592.034915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828592.034930:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.034938:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb220700 x1874663333156352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.034945:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333156352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.034951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.034952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.034954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828592.034991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828592.035004:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.035010:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb220380 x1874663333156480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.035023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333156480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.035028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.035030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.035031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828592.035045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828592.035055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.035061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb222300 x1874663333156608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.035068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333156608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.035074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.035076:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.035077:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333156608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828592.035081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828592.035091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.035096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb221c00 x1874663333156736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.035103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333156736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.035122:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.035124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.035126:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828592.035130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828592.035141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.035147:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb222d80 x1874663333156864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.035153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333156864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.035158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.035160:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.035162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828592.035167:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828592.035177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828592.035183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9057fb222a00 x1874663333156992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828592.035189:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333156992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828592.035204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828592.035206:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828592.035208:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333156992:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828592.418393:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828593.442453:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828594.466491:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828595.490497:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828596.514437:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828597.154298:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154308:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828597.154313:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154323:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154334:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154340:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828597.154371:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333157120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154394:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154397:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.154400:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154404:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154408:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154412:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154422:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.154436:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333157248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154446:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154449:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.154451:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154455:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154462:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154465:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.154475:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333157376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828597.154491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333157120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828597.154494:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.154514:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154518:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154520:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154523:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828597.154536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333157504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154543:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.154549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154551:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154554:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154556:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828597.154567:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333157632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.154577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154580:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154583:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154586:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154591:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828597.154599:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828597.154620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333157760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154630:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.154634:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154639:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154644:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154648:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154653:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.154663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333157888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.154671:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.154675:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828597.154676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158e680 x1874663333157120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828597.154678:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.154690:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10407 00000100:00080000:0.0:1787828597.154701:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.154710:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828597.154715:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828597.154722:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333157120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828597.154726:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333158016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828597.154767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.154782:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.154788:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333157120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828597.154802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.154829:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.154844:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158f800 x1874663333157248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.154859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333157248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.154866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.154868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.154870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.154896:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.154913:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.154921:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158c000 x1874663333157376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.154929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333157376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.154934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.154936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.154938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.154942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.154954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.154960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158dc00 x1874663333157504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.154967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333157504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.154972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.154974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.154976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828597.154979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.154989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.154995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158e300 x1874663333157632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.155007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333157632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.155013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.155015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.155017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333157632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828597.155020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333157760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.155030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.155036:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158f100 x1874663333157760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.155042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333157760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.155048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.155050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.155052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333157760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.155055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333157888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.155065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.155071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158ca80 x1874663333157888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.155077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333157888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.155087:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.155090:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.155092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333157888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.155095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.155106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828597.155111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9057f158df80 x1874663333158016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.155118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333158016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.155124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.155126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828597.155128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158016:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828597.538369:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828598.562427:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828599.586383:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828600.610389:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828601.634375:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828602.274353:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274363:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828602.274367:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828602.274377:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274387:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828602.274394:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828602.274430:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333158144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828602.274457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274460:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828602.274463:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828602.274468:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274473:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828602.274477:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828602.274500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333158272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828602.274506:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333158144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828602.274521:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274525:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828602.274527:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828602.274532:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828602.274541:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274546:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828602.274559:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333158400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828602.274566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828602.274589:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274593:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828602.274595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828602.274601:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274606:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828602.274611:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274617:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828602.274619:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb222680 x1874663333158144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828602.274633:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333158528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828602.274636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333158144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828602.274662:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828602.274669:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.274675:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333158144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828602.274680:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828602.274687:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828602.274692:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274696:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828602.274698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828602.274703:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274710:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828602.274721:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828602.274726:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333158656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828602.274734:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb223800 x1874663333158272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828602.274744:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828602.274749:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333158272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828602.274751:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828602.274755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828602.274760:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828602.274765:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:2.0:1787828602.274766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.274772:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828602.274774:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828602.274779:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158400:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828602.274784:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274791:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828602.274806:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333158784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828602.274810:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828602.274821:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb221500 x1874663333158400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828602.274822:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274832:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828602.274838:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828602.274842:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333158400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828602.274847:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:2.0:1787828602.274857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828602.274857:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828602.274863:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.274867:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158400:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828602.274867:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828602.274879:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828602.274883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828602.274894:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333158912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828602.274903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828602.274909:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828602.274914:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828602.274917:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828602.274919:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb221180 x1874663333158528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828602.274923:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10413 00000100:00080000:3.0:1787828602.274931:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828602.274938:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828602.274943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333158528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828602.274945:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828602.274959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828602.274963:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333159040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828602.274976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.274981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828602.274998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828602.275015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828602.275023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb220000 x1874663333158656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828602.275031:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333158656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828602.275037:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828602.275039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.275041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333158656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828602.275044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828602.275055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828602.275061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb223b80 x1874663333158784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828602.275073:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333158784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828602.275078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828602.275079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.275081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828602.275084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828602.275094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828602.275100:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb220a80 x1874663333158912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828602.275106:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333158912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828602.275111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828602.275113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.275115:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333158912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828602.275118:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828602.275127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828602.275133:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828602/real 1787828602] req@ffff9057fb220e00 x1874663333159040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828602.275139:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333159040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828602.275149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828602.275151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828602.275153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159040:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828602.658406:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828603.682433:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828604.706360:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828605.730450:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828606.754387:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828607.394305:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394313:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828607.394316:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394325:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394334:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394341:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828607.394377:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333159168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394403:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394406:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828607.394407:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394410:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394414:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394417:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394425:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828607.394435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333159296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394442:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394444:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828607.394446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394455:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828607.394461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333159424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394470:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828607.394471:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394474:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394476:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394479:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394482:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828607.394489:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333159168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828607.394490:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333159552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828607.394506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394509:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394511:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394513:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828607.394525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333159680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828607.394552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394555:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394559:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394561:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828607.394567:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828607.394580:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333159808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828607.394591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394594:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394597:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394600:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394603:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828607.394605:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158df80 x1874663333159168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828607.394613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333159936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828607.394622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828607.394626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828607.394626:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333159168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828607.394630:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828607.394638:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10418 00000100:00080000:0.0:1787828607.394643:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828607.394646:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828607.394648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828607.394654:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828607.394655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333160064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828607.394664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394671:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333159168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828607.394678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828607.394704:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394718:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158ca80 x1874663333159296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333159296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394737:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394739:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394742:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828607.394747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828607.394761:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158f100 x1874663333159424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394776:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333159424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394781:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394783:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394785:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828607.394803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828607.394814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158e300 x1874663333159552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333159552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828607.394839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828607.394849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158dc00 x1874663333159680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333159680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333159680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828607.394873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828607.394882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158c000 x1874663333159808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333159808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394901:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828607.394906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828607.394915:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394921:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158f800 x1874663333159936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394928:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333159936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394933:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333159936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828607.394940:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828607.394950:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828607.394955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828607/real 1787828607] req@ffff9057f158e680 x1874663333160064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828607.394961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333160064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828607.394966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828607.394968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828607.394970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160064:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828607.778449:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828608.802348:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828609.826451:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828610.850341:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828611.874465:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828612.514370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514381:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828612.514385:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514396:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514407:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828612.514469:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333160192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828612.514495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828612.514512:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514516:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514519:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514523:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514533:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828612.514551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333160320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828612.514559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828612.514564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514567:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514569:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514572:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828612.514583:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333160192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828612.514589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333160448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828612.514599:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828612.514604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514607:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514608:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514611:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828612.514624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333160576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828612.514633:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514637:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828612.514639:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514643:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514650:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514654:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828612.514667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828612.514684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333160704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828612.514696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828612.514701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514705:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514710:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514714:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514720:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828612.514733:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333160832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828612.514735:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78a8a80 x1874663333160192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828612.514747:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828612.514761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828612.514766:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514779:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00080000:1.0:1787828612.514785:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514789:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514793:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828612.514808:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333160960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828612.514810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333160192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.514873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828612.514876:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828612.514880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828612.514888:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828612.514888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333160192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828612.514897:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828612.514906:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10423 00000100:00100000:2.0:1787828612.514906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160320:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828612.514916:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828612.514922:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828612.514928:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828612.514940:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333161088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828612.514951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.514979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78ab100 x1874663333160320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.514994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333160320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515006:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828612.515026:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828612.515042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78a9500 x1874663333160448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333160448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515063:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515065:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828612.515070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828612.515082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515088:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78a8700 x1874663333160576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515096:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333160576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828612.515108:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828612.515118:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515124:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78a9180 x1874663333160704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515131:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333160704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515138:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515140:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333160704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828612.515143:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828612.515154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78aa300 x1874663333160832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515172:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333160832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515179:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515180:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828612.515184:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828612.515194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515199:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78a9c00 x1874663333160960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515206:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333160960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515212:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515214:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333160960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828612.515258:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828612.515273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828612.515280:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828612/real 1787828612] req@ffff9057c78aa680 x1874663333161088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828612.515288:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333161088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828612.515297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828612.515299:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828612.515301:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161088:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828612.898394:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828613.922472:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828614.946377:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828615.970446:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828616.994432:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828617.634337:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634347:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828617.634352:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634380:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828617.634416:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333161216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634441:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828617.634445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634451:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828617.634461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828617.634486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333161344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634494:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634497:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828617.634498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634502:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634509:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828617.634511:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333161216:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828617.634514:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828617.634524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333161472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634534:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828617.634536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634538:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634540:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828617.634546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828617.634554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333161600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828617.634563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634571:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828617.634575:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828617.634580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828617.634585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333161728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634593:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828617.634597:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634602:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634604:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828617.634607:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828617.634616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333161856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828617.634624:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158d880 x1874663333161216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828617.634625:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634630:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828617.634634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634640:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:0.0:1787828617.634644:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634648:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828617.634651:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828617.634659:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333161216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828617.634660:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333161984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828617.634688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828617.634692:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828617.634693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828617.634694:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828617.634702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10428 00000100:00080000:2.0:1787828617.634703:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828617.634705:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828617.634710:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828617.634714:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333161216:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828617.634715:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828617.634726:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333162112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828617.634729:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828617.634767:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.634783:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158ce00 x1874663333161344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.634798:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333161344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.634806:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.634810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.634815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828617.634838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828617.634861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.634873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158c700 x1874663333161472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.634887:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333161472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.634896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.634899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.634903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828617.634909:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828617.634928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.634940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158d180 x1874663333161600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.634953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333161600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.634962:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.634965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.634969:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828617.634975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828617.634995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.635007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158d500 x1874663333161728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.635020:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333161728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.635029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.635032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.635036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333161728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828617.635043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828617.635062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.635072:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158ed80 x1874663333161856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.635085:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333161856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.635094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.635098:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.635101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828617.635108:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828617.635128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.635140:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158fb80 x1874663333161984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.635152:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333161984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.635161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.635164:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.635167:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333161984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828617.635173:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333162112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828617.635192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828617.635204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828617] req@ffff9057f158ea00 x1874663333162112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828617.635217:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333162112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828617.635254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828617.635257:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828617.635260:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333162112:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828618.018450:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828619.042351:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828620.066377:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828621.090354:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828622.114353:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828622.754283:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754294:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828622.754298:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828622.754310:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754321:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754330:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828622.754364:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333162240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828622.754385:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754388:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828622.754389:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828622.754393:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754396:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754399:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828622.754407:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828622.754416:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333162368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828622.754425:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828622.754439:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333162240:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828622.754443:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828622.754448:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828622.754451:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754454:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754457:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828622.754461:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828622.754471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333162496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828622.754478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754481:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828622.754482:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828622.754485:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754488:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754490:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828622.754494:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828622.754499:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828622.754503:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333162624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828622.754511:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754514:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828622.754515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828622.754518:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754520:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754522:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828622.754526:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828622.754534:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158ea00 x1874663333162240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.754537:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333162752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828622.754559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754562:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828622.754563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828622.754565:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333162240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828622.754571:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754579:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828622.754579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754586:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828622.754590:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754593:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333162240:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828622.754599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828622.754615:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333162880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828622.754615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162368:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828622.754631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828622.754634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828622.754636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828622.754641:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828622.754644:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00080000:0.0:1787828622.754649:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754653:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754657:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158fb80 x1874663333162368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828622.754660:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828622.754672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333162368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.754676:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333163008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828622.754693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.754698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828622.754700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828622.754702:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828622.754707:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828622.754708:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162368:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828622.754716:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10433 00000100:00100000:2.0:1787828622.754719:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162496:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828622.754722:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828622.754727:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828622.754730:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828622.754739:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333163136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828622.754747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754759:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158ed80 x1874663333162496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.754767:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333162496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.754773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754775:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.754777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828622.754781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828622.754794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158d500 x1874663333162624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.754807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333162624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.754812:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.754815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828622.754818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828622.754828:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158d180 x1874663333162752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.754841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333162752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.754846:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.754849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333162752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828622.754860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333162880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828622.754870:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158c700 x1874663333162880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.754891:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333162880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.754899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.754906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333162880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828622.754912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828622.754932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.754944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158ce00 x1874663333163008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.754956:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333163008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.754965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.754969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.754972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828622.754979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828622.754997:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828622.755004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff9057f158d880 x1874663333163136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828622.755012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333163136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828622.755022:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828622.755024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828622.755026:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163136:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828623.138356:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828624.162394:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828625.186402:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828626.210480:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828627.234452:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828627.874301:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874312:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828627.874315:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874327:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874339:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874346:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828627.874388:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333163264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874428:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874431:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828627.874433:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874437:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874440:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874443:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874453:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828627.874464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333163392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874474:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874477:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828627.874478:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874481:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874483:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874485:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874488:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828627.874495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333163520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874504:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828627.874508:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333163264:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828627.874509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828627.874514:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874527:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874530:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874533:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874539:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828627.874550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333163648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828627.874563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874567:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874577:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874581:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828627.874588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828627.874603:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333163776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874611:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874614:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828627.874615:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874622:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874627:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874631:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828627.874636:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb220e00 x1874663333163264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828627.874647:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333163904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828627.874658:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828627.874659:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333163264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828627.874666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828627.874671:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874683:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00080000:3.0:1787828627.874690:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828627.874693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828627.874693:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828627.874707:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828627.874712:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333164032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828627.874722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333163264:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828627.874726:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828627.874732:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828627.874734:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828627.874738:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10438 00000100:00100000:2.0:1787828627.874741:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163392:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828627.874749:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828627.874753:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828627.874757:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828627.874770:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333164160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828627.874771:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.874787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb220a80 x1874663333163392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.874800:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333163392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.874807:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.874810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.874813:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828627.874834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828627.874846:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.874853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb223b80 x1874663333163520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.874861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333163520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.874866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.874868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.874869:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828627.874873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828627.874884:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.874890:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb220000 x1874663333163648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.874897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333163648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.874902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.874904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.874906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828627.874910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828627.874920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.874926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb221180 x1874663333163776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.874934:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333163776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.874942:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.874945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.874949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333163776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828627.874955:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828627.874976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.874987:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb221500 x1874663333163904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.875000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333163904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.875008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.875011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.875015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333163904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828627.875022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828627.875043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.875052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb223800 x1874663333164032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.875060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333164032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.875065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.875068:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.875070:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828627.875074:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828627.875086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828627.875092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff9057fb222680 x1874663333164160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.875099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333164160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828627.875112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828627.875114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828627.875116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164160:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828628.258439:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828629.282456:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828630.306363:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828631.330560:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828632.354469:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828632.994314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828632.994329:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994341:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994354:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828632.994422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333164288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828632.994447:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994450:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828632.994451:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994455:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994458:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828632.994479:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333164416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828632.994487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828632.994493:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994498:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994502:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994505:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828632.994514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333164544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828632.994521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828632.994525:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994528:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994531:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994533:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828632.994546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333164672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828632.994546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333164288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828632.994570:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828632.994573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994576:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994580:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994583:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994588:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828632.994599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333164800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828632.994607:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828632.994611:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994615:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994617:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994621:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994624:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828632.994633:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828632.994637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333164928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828632.994646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828632.994651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994654:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994656:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828632.994659:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828632.994666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828632.994680:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333165056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828632.994685:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78aaa00 x1874663333164288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828632.994703:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828632.994718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828632.994723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828632.994728:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10443 00000100:00080000:1.0:1787828632.994732:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828632.994732:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333164288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828632.994744:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828632.994755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828632.994759:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828632.994763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.994773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333164288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828632.994779:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333165184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828632.994783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828632.994824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.994837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78ab480 x1874663333164416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.994845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333164416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.994850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.994852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.994854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828632.994859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828632.994870:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.994876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78abb80 x1874663333164544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.994883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333164544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.994887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.994889:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.994891:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828632.994910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828632.994921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.994927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78a8000 x1874663333164672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.994935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333164672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.994940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.994941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.994943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828632.994947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828632.994957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.994963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78aad80 x1874663333164800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.994970:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333164800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.994974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.994976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.994978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333164800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828632.994982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828632.994991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.994997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78a9880 x1874663333164928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.995008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333164928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.995013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.995015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.995017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333164928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828632.995020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828632.995030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.995035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78ab800 x1874663333165056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.995041:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333165056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.995046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.995048:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.995050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828632.995053:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828632.995063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828632.995069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff9057c78a8380 x1874663333165184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.995075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333165184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.995084:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828632.995086:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828632.995088:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165184:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828633.378441:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828634.402448:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828635.426443:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828636.450560:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828637.474448:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828638.114330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114351:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828638.114354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114363:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828638.114379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828638.114410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333165312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828638.114439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828638.114443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828638.114451:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114459:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828638.114468:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333165440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828638.114477:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114480:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828638.114484:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828638.114485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333165312:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828638.114497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114503:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828638.114506:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828638.114521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333165568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828638.114529:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828638.114533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114536:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114548:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828638.114554:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828638.114559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828638.114583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333165696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.114597:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb222a00 x1874663333165312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828638.114597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828638.114610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114635:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828638.114646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333165312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828638.114646:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114657:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828638.114663:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.114669:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828638.114674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333165824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.114676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333165312:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828638.114690:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828638.114695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114698:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114701:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828638.114708:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165440:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828638.114708:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828638.114730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333165952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828638.114749:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114751:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828638.114753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114756:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10448 00000800:00000100:2.0:1787828638.114757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828638.114759:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828638.114765:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114770:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828638.114780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333166080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.114786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb222d80 x1874663333165440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828638.114792:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828638.114802:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828638.114806:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828638.114812:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10448 00000100:00080000:3.0:1787828638.114815:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828638.114817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333165440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828638.114821:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828638.114848:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828638.114851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.114859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.114872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.114883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828638.114883:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333166208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828638.114916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.114931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb221c00 x1874663333165568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.114944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333165568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.114951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.114954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.114956:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.114961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.114986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.114995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb222300 x1874663333165696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.115004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333165696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.115011:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.115014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.115017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828638.115021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.115035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.115043:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb220380 x1874663333165824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.115052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333165824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.115059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.115061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.115064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333165824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828638.115079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828638.115093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.115101:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb220700 x1874663333165952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.115111:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333165952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.115117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.115119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.115122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333165952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.115126:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828638.115139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.115162:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb221880 x1874663333166080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.115170:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333166080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.115183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.115196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.115199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.115203:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.115216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828638.115272:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9057fb223100 x1874663333166208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.115283:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333166208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.115290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.115293:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828638.115296:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166208:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828638.498566:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828639.522410:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828640.546423:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828641.570374:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828642.594395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828643.234310:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234320:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828643.234325:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234337:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234349:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234357:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828643.234399:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333166336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234425:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234428:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828643.234429:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234432:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234435:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234437:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234446:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828643.234456:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333166464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234463:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234466:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828643.234468:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234470:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234473:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234475:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234478:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828643.234486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333166592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234492:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828643.234497:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234501:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234505:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234509:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828643.234516:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333166336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828643.234524:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333166720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234534:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234536:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828643.234537:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234546:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828643.234557:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333166848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234563:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234565:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828643.234568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234571:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234590:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234595:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234601:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828643.234616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333166976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828643.234629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828643.234635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828643.234641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166464:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828643.234641:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234652:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828643.234658:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828643.234680:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333167104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828643.234682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166592:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828643.234704:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828643.234707:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828643.234710:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828643.234714:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10453 00000100:00080000:3.0:1787828643.234720:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828643.234724:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166720:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828643.234725:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828643.234740:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828643.234752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333167232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828643.234759:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828643.234795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828643.234818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333167104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828643.234836:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333167232:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828643.618366:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828644.642424:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828645.666530:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828646.306681:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787828646.306740:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828646.306760:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306798:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306808:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306820:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306847:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828646.306857:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb220380 x1874663333166720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828646.306859:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306871:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828646.306882:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828646.306909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333166720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.306922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.306927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.306932:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828646.306940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb222300 x1874663333166848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.306954:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333166848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.306962:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.306965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.306969:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828646.306977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb221c00 x1874663333166976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.306991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333166976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333166976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.307013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb222d80 x1874663333167104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.307024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333167104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307035:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333167104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.307045:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb223100 x1874663333166336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.307056:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333166336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333166336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828646.307078:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb221880 x1874663333166464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.307090:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333166464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307098:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307105:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.307113:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb220700 x1874663333166592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.307126:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333166592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307138:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333166592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828646.307174:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828643/real 1787828646] req@ffff9057fb222a00 x1874663333167232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828646.307186:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333167232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828646.307201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828646.307204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828646.307208:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333167232:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828646.690430:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828647.714374:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828648.354339:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354350:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828648.354355:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354367:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354379:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828648.354387:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828648.354431:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333167360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828648.354454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354459:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828648.354460:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354465:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828648.354471:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828648.354480:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828648.354483:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333167360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828648.354496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333167488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828648.354513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828648.354520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354541:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828648.354548:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828648.354556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828648.354561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828648.354581:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333167616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828648.354597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354601:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828648.354602:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354609:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354613:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828648.354619:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828648.354622:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb222680 x1874663333167360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828648.354625:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828648.354642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333167744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828648.354654:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828648.354657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333167360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828648.354660:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828648.354665:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828648.354682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828648.354684:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354694:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828648.354696:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828648.354709:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828648.354711:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333167360:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828648.354719:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828648.354740:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333167872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828648.354753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167488:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828648.354754:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354765:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828648.354769:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354779:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354786:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828648.354793:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828648.354801:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828648.354809:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828648.354824:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333168000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828648.354838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb223800 x1874663333167488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828648.354845:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354857:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828648.354863:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354883:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.354893:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828648.354900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333167488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828648.354902:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828648.354914:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828648.354920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.354929:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828648.354936:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333168128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828648.354938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828648.354968:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167616:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828648.354970:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828648.354984:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828648.354987:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828648.354993:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10459 00000100:00080000:3.0:1787828648.355010:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828648.355011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828648.355022:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828648.355031:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828648.355035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb221500 x1874663333167616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828648.355054:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333168256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828648.355057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333167616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355094:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355098:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828648.355105:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828648.355130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828648.355142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb221180 x1874663333167744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828648.355154:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333167744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355167:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355170:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828648.355190:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828648.355213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828648.355276:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb220000 x1874663333167872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828648.355288:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333167872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355299:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355303:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333167872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828648.355309:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828648.355328:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828648.355340:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb223b80 x1874663333168000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828648.355352:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333168000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355363:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355367:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828648.355373:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828648.355394:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828648.355406:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb220a80 x1874663333168128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828648.355418:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333168128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355431:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355435:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828648.355833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828648.355914:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828648.355944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828648/real 1787828648] req@ffff9057fb220e00 x1874663333168256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828648.355959:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333168256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828648.355968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828648.355971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828648.355974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333168256:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828648.738447:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828649.762491:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828650.786459:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828651.810412:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828652.834498:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828653.474299:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474307:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828653.474310:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474319:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474328:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474334:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828653.474365:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333168384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474386:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474389:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828653.474390:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474393:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474396:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474398:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828653.474405:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828653.474413:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333168512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474421:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474424:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828653.474425:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474427:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474429:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474431:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828653.474437:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828653.474444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333168640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474452:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828653.474455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474457:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474461:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474464:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828653.474468:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828653.474486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333168768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828653.474493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333168384:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828653.474500:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828653.474512:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474516:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474521:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474526:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828653.474531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828653.474541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333168896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474552:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828653.474554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474558:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474575:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474581:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828653.474588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828653.474593:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828653.474613:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333169024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474622:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828653.474625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828653.474626:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474633:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474637:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828653.474642:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474646:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb220e00 x1874663333168384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828653.474647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828653.474663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333169152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828653.474670:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828653.474673:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333168384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828653.474675:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828653.474682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828653.474697:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10464 00000100:00080000:3.0:1787828653.474705:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828653.474706:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828653.474715:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828653.474718:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828653.474724:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828653.474733:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333168384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828653.474741:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333169280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828653.474745:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828653.474783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb220a80 x1874663333168512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.474813:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333168512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.474819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.474822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.474824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828653.474828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828653.474839:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474845:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb223b80 x1874663333168640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.474853:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333168640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.474858:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.474860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.474862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828653.474887:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828653.474899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb220000 x1874663333168768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.474912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333168768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.474922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.474924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.474927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828653.474930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828653.474940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb221180 x1874663333168896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.474953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333168896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.474959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.474960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.474962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333168896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828653.474966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828653.474975:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.474981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb221500 x1874663333169024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.474989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333169024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.474994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.474995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.474998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828653.475002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828653.475013:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.475019:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb223800 x1874663333169152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.475026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333169152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.475031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.475033:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.475035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828653.475038:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828653.475049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828653.475054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828653/real 1787828653] req@ffff9057fb222680 x1874663333169280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828653.475061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333169280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828653.475066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828653.475068:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828653.475070:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333169280:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828653.858431:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828654.882418:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828655.906414:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828656.930535:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828657.954380:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828658.594315:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594323:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828658.594326:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594365:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828658.594398:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333169408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.594422:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594425:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828658.594426:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594430:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594435:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594442:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.594451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333169536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.594460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.594465:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594467:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594470:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594474:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594487:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.594495:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333169664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828658.594503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333169408:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828658.594506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828658.594516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594519:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594521:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594524:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828658.594536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333169792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.594551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594554:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.594555:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594558:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594577:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594581:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828658.594592:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333169920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.594598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828658.594602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828658.594607:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828658.594608:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594613:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594616:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594619:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.594626:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333170048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.594633:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594635:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.594636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594641:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.594643:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594655:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.594663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333170176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828658.594669:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb222a00 x1874663333169408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828658.594671:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.594684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.594686:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.594690:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10469 00000100:00080000:3.0:1787828658.594693:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828658.594696:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333169408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828658.594697:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828658.594704:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828658.594715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333170304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828658.594728:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.594737:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.594743:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333169408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828658.594753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.594779:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.594793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb220700 x1874663333169536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.594807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333169536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.594816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.594820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.594823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.594848:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.594871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.594884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb221880 x1874663333169664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.594906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333169664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.594915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.594918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.594922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.594929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828658.594949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.594961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb223100 x1874663333169792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.594974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333169792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.594983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.594986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.594989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828658.595005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828658.595025:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.595037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb222d80 x1874663333169920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.595050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333169920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.595077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.595081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.595085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333169920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828658.595092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.595111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.595123:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb221c00 x1874663333170048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.595136:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333170048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.595145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.595148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.595152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.595159:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.595196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.595208:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb222300 x1874663333170176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.595216:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333170176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.595243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.595245:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.595248:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.595252:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828658.595277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828658.595284:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff9057fb220380 x1874663333170304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.595291:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333170304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.595296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.595298:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828658.595300:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333170304:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828658.978395:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828660.002437:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828661.026424:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828662.050567:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828663.074502:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828663.714298:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714310:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828663.714315:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.714325:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714337:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714344:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828663.714381:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333170432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.714409:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714413:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828663.714415:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.714420:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714425:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714429:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828663.714439:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828663.714455:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333170560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828663.714461:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333170432:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828663.714474:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714479:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828663.714481:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.714487:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714492:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828663.714503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828663.714517:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333170688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828663.714521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828663.714533:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828663.714540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.714545:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714548:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714553:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828663.714561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828663.714565:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb220380 x1874663333170432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828663.714577:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333170816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828663.714583:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333170432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828663.714606:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828663.714612:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828663.714613:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828663.714618:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828663.714620:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333170432:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828663.714628:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714634:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714639:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714644:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170560:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828663.714648:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828663.714664:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333170944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828663.714674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714686:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb222300 x1874663333170560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714691:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714698:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828663.714701:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333170560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828663.714713:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828663.714714:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:2.0:1787828663.714721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828663.714723:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828663.714724:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170560:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828663.714733:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714734:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170688:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828663.714744:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828663.714756:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828663.714761:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333171072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.714773:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.714778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828663.714780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb221c00 x1874663333170688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714781:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.714790:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00100000:2.0:1787828663.714804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333170688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714807:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714816:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828663.714819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828663.714824:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828663.714825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.714831:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828663.714837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828663.714844:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333171200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.714856:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828663.714859:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828663.714863:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828663.714867:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828663.714872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb222d80 x1874663333170816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.714873:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10474 00000100:00080000:3.0:1787828663.714880:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.714889:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714898:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333170816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828663.714898:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828663.714909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.714911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.714914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828663.714918:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333171328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828663.714931:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828663.714946:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb223100 x1874663333170944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.714962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333170944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.714967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.714969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.714971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333170944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828663.714974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828663.714984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828663.714989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb221880 x1874663333171072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.714996:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333171072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.715005:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.715007:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.715009:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828663.715012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828663.715021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828663.715026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb220700 x1874663333171200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.715033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333171200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.715037:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.715039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.715041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828663.715049:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828663.715059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828663.715066:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff9057fb222a00 x1874663333171328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.715073:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333171328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.715077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.715079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828663.715081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333171328:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828664.098392:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828665.122387:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828666.146583:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828667.170457:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828668.194426:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828668.834369:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834379:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828668.834384:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834394:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834407:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828668.834462:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333171456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834501:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828668.834511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834515:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834518:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834522:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834532:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828668.834541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333171584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828668.834556:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333171456:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828668.834557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834564:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834567:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834569:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828668.834583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333171712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834590:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828668.834605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834608:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834610:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834612:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834616:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828668.834623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333171840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828668.834633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834636:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834641:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828668.834647:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828668.834655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333171968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828668.834668:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828668.834671:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834673:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828668.834686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333172096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828668.834692:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828668.834698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828668.834699:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb222680 x1874663333171456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828668.834704:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834709:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834716:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834721:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828668.834728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333171456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828668.834731:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333172224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828668.834752:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828668.834752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828668.834759:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828668.834762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828668.834763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828668.834769:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333171456:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828668.834769:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10479 00000100:00080000:3.0:1787828668.834775:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828668.834779:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828668.834782:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828668.834788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333172352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828668.834803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.834856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.834876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb223800 x1874663333171584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.834899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333171584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.834907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.834910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.834913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.834919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.834968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.834981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb221500 x1874663333171712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333171712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.835029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828668.835046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.835054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb221180 x1874663333171840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835062:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333171840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828668.835075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828668.835085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.835091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb220000 x1874663333171968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333171968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333171968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828668.835142:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.835157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.835164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb223b80 x1874663333172096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835172:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333172096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835179:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835181:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.835195:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.835205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.835211:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb220a80 x1874663333172224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835217:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333172224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835271:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835274:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.835278:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828668.835292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828668.835299:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff9057fb220e00 x1874663333172352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.835307:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333172352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.835312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.835314:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828668.835316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333172352:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828669.218427:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828670.242461:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828671.266403:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828672.290448:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828673.314426:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828673.954325:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954335:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828673.954339:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954349:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954360:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954376:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828673.954414:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333172480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828673.954454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828673.954477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828673.954492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333172608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828673.954511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828673.954513:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333172480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828673.954519:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954526:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954529:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828673.954534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828673.954549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333172736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828673.954565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954573:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954577:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828673.954584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828673.954588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828673.954600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333172864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828673.954618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954623:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954626:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954631:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828673.954636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828673.954645:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158e680 x1874663333172480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828673.954652:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333172992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828673.954677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954682:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954686:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828673.954690:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333172480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828673.954694:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828673.954703:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828673.954711:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828673.954720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333173120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828673.954721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.954731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333172480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828673.954740:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828673.954748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954754:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10484 00000100:00080000:0.0:1787828673.954758:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954763:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828673.954767:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172608:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828673.954770:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828673.954787:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333173248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828673.954799:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828673.954803:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828673.954805:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.954809:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10484 00000800:00000100:2.0:1787828673.954815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828673.954826:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828673.954832:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828673.954836:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158f800 x1874663333172608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828673.954840:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828673.954860:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333173376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828673.954863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333172608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.954891:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.954896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.954900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828673.954907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828673.954932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.954946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158c000 x1874663333172736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.954960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333172736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.954970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.954973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.954977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828673.954984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828673.955006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.955018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158dc00 x1874663333172864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.955032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333172864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.955041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.955044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.955048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828673.955055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828673.955094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.955107:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158e300 x1874663333172992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.955120:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333172992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.955129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.955133:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.955136:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333172992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828673.955143:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828673.955163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.955176:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158f100 x1874663333173120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.955196:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333173120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.955204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.955208:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.955211:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828673.955259:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828673.955282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.955294:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158ca80 x1874663333173248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.955308:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333173248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.955317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.955320:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.955324:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828673.955331:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828673.955350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828673.955362:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff9057f158df80 x1874663333173376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.955374:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333173376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.955390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.955393:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828673.955397:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333173376:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828674.338496:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828675.362363:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828676.386337:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828677.410419:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828678.434443:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828679.074329:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828679.074354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074363:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074374:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074381:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828679.074438:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333173504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074490:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828679.074497:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074503:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074509:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074515:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828679.074526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828679.074557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333173632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828679.074574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074578:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074582:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828679.074590:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333173504:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828679.074594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828679.074617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333173760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074631:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828679.074633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074636:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074643:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828679.074648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828679.074656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333173888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074667:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828679.074670:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828679.074674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828679.074678:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074683:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074687:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828679.074691:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828679.074702:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333174016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074709:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074713:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828679.074715:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074719:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074722:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074725:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828679.074730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828679.074734:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158df80 x1874663333173504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828679.074741:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333174144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828679.074775:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074780:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828679.074780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333173504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828679.074783:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074798:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074804:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074812:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828679.074815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828679.074817:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828679.074826:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828679.074831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333174272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828679.074833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333173504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828679.074848:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173632:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828679.074861:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828679.074864:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828679.074866:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828679.074871:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10489 00000100:00080000:0.0:1787828679.074875:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828679.074878:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828679.074883:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828679.074886:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828679.074901:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158ca80 x1874663333173632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828679.074903:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333174400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828679.074944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333173632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.074953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.074957:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.074960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.074988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828679.075003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158f100 x1874663333173760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075021:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333173760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.075057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.075076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075094:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158e300 x1874663333173888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075104:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333173888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333173888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828679.075121:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.075133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158dc00 x1874663333174016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075150:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333174016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075159:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828679.075166:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828679.075178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075185:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158c000 x1874663333174144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075216:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333174144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075253:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075256:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.075261:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828679.075290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075299:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158f800 x1874663333174272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075309:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333174272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075319:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075321:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.075325:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.075365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828679.075373:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9057f158e680 x1874663333174400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.075383:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333174400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.075390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.075392:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828679.075395:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333174400:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828679.458375:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828680.482396:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828681.506537:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828682.530363:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828683.554494:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828684.194350:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194361:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828684.194367:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194378:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194390:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828684.194399:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828684.194455:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333174528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828684.194485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828684.194509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194518:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194527:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828684.194536:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828684.194557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333174528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828684.194581:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333174656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828684.194605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828684.194615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828684.194631:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828684.194648:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828684.194657:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333174784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828684.194689:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828684.194695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194700:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194705:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828684.194706:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158d880 x1874663333174528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828684.194712:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194723:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828684.194729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333174528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.194739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828684.194741:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333174912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828684.194745:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.194753:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333174528:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828684.194758:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194762:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828684.194763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194767:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194770:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828684.194773:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194777:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828684.194777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828684.194794:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333175040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828684.194803:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828684.194807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828684.194808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828684.194812:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194816:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.194821:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828684.194824:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158ce00 x1874663333174656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828684.194829:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194841:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828684.194843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333174656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.194852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.194854:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.194861:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828684.194877:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333175168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828684.194893:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828684.194914:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828684.194919:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828684.194922:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.194929:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10494 00000800:00000100:2.0:1787828684.194936:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828684.194937:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828684.194947:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.194956:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828684.194957:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158c700 x1874663333174784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.194975:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333174784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828684.194986:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333175296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828684.194989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828684.195060:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828684.195068:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174912:192.168.203.156@tcp:8:kworker.0 00000100:00080000:0.0:1787828684.195071:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828684.195078:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828684.195087:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10494 00000100:00080000:0.0:1787828684.195095:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828684.195103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828684.195110:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828684.195124:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828684.195135:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158d180 x1874663333174912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828684.195152:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333175424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828684.195167:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333174912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.195198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195203:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333174912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828684.195244:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828684.195301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828684.195334:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158d500 x1874663333175040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.195363:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333175040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.195375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195380:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195383:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828684.195390:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828684.195409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828684.195449:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158ed80 x1874663333175168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.195465:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333175168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.195471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195473:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195475:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828684.195478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828684.195490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828684.195497:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158fb80 x1874663333175296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.195523:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333175296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.195531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195534:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195537:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828684.195542:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828684.195559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828684.195567:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9057f158ea00 x1874663333175424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.195578:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333175424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.195592:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.195595:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828684.195598:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333175424:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828684.578377:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828685.602438:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828686.626506:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828687.650413:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828688.674402:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828689.314331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828689.314359:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314376:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828689.314432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333175552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314456:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828689.314458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828689.314483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333175680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828689.314499:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314501:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314503:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314506:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828689.314533:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333175808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314540:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314542:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828689.314544:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314546:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314548:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314550:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314554:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828689.314558:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333175552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828689.314563:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333175936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314572:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314574:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828689.314575:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314579:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314582:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828689.314599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333176064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828689.314608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314611:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314615:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314618:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828689.314628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828689.314630:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333176192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828689.314641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314644:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00080000:3.0:1787828689.314646:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314649:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828689.314659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333176320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828689.314670:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828689.314674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828689.314676:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828689.314680:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10500 00000100:00100000:2.0:1787828689.314683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb220e00 x1874663333175552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828689.314684:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828689.314690:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828689.314698:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828689.314716:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333176448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828689.314719:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333175552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314744:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314752:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333175552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828689.314759:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828689.314780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.314790:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb220a80 x1874663333175680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.314798:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333175680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828689.314813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828689.314823:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.314829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb223b80 x1874663333175808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.314836:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333175808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314843:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828689.314863:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828689.314874:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.314880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb220000 x1874663333175936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.314888:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333175936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333175936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828689.314899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828689.314908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.314915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb221180 x1874663333176064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.314921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333176064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828689.314933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828689.314943:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.314948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb221500 x1874663333176192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.314956:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333176192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.314960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.314962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.314980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828689.314985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828689.314998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.315005:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb223800 x1874663333176320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.315012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333176320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.315018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.315020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.315022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828689.315025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828689.315035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828689.315040:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828689/real 1787828689] req@ffff9057fb222680 x1874663333176448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828689.315047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333176448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828689.315052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828689.315054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828689.315056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333176448:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828689.698402:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828690.722417:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828691.746462:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828692.770355:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828693.794424:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828694.434386:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434398:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828694.434403:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434415:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434428:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434437:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828694.434484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333176576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.434522:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434527:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828694.434529:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434535:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434539:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434543:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434554:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828694.434571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333176704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.434588:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828694.434595:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333176576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828694.434596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828694.434604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434615:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434619:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828694.434645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333176832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.434659:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434664:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828694.434666:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434670:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434675:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434679:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434686:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828694.434694:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828694.434707:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333176960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.434721:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434724:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828694.434726:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434733:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434739:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434744:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434750:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828694.434755:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158ea00 x1874663333176576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828694.434770:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333177088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.434807:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434811:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828694.434814:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434823:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00100000:2.0:1787828694.434825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333176576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828694.434828:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434837:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434848:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828694.434872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828694.434874:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333177216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828694.434893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828694.434899:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.434906:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828694.434912:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333176576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828694.434913:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.434958:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.434966:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.434972:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828694.434979:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828694.434999:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828694.435001:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333177344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828694.435029:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828694.435033:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828694.435037:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828694.435044:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10505 00000100:00080000:0.0:1787828694.435050:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828694.435056:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828694.435064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828694.435069:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828694.435090:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333177472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828694.435096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158fb80 x1874663333176704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333176704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435167:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828694.435183:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828694.435207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435261:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158ed80 x1874663333176832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435282:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333176832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435299:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435306:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435313:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828694.435325:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828694.435368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435412:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158d500 x1874663333176960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435428:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333176960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435443:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435447:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333176960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828694.435456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828694.435482:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435495:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158d180 x1874663333177088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435509:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333177088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435519:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435522:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435526:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828694.435532:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828694.435552:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435563:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158c700 x1874663333177216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435587:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333177216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435600:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435603:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828694.435629:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828694.435649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435660:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158ce00 x1874663333177344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435674:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333177344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435686:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435690:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828694.435696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828694.435715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828694.435725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828694/real 1787828694] req@ffff9057f158d880 x1874663333177472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828694.435744:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333177472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828694.435773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828694.435779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828694.435786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333177472:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828694.818455:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828695.842540:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828696.866443:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828697.890509:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828698.914445:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828699.554305:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554313:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828699.554317:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554325:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554333:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554339:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828699.554372:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333177600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828699.554394:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554397:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828699.554398:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554401:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554404:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554406:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554417:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828699.554427:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333177728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828699.554436:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554439:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828699.554440:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554443:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554445:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554447:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554451:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828699.554455:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333177600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828699.554461:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333177856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828699.554470:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554473:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828699.554474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554477:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554479:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554481:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828699.554494:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333177984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828699.554501:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554503:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828699.554504:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554507:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554510:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554512:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554515:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828699.554525:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333178112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828699.554525:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828699.554535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828699.554539:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554543:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554545:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554548:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828699.554558:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333178240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828699.554564:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554567:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828699.554568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554573:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00100000:2.0:1787828699.554574:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158e680 x1874663333177600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828699.554576:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828699.554581:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828699.554598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333178368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828699.554608:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333177600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828699.554613:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828699.554620:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828699.554622:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828699.554629:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10510 00000100:00080000:0.0:1787828699.554632:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828699.554633:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828699.554634:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828699.554637:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828699.554642:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333178496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828699.554653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.554661:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333177600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828699.554671:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828699.554698:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.554712:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158f800 x1874663333177728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.554745:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333177728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.554757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.554761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.554767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828699.554792:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828699.554811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.554838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158c000 x1874663333177856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.554849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333177856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.554856:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.554859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.554864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828699.554872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828699.554934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.554950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158dc00 x1874663333177984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.554966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333177984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.554975:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.554979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.554983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333177984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828699.554991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828699.555017:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.555032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158e300 x1874663333178112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.555048:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333178112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.555059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.555063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.555068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828699.555076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828699.555098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.555111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158f100 x1874663333178240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.555128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333178240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.555152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.555155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.555158:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828699.555165:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828699.555185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.555196:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158ca80 x1874663333178368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.555208:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333178368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.555218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.555221:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.555224:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828699.555230:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828699.555285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828699.555296:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828699/real 1787828699] req@ffff9057f158df80 x1874663333178496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828699.555308:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333178496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828699.555317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828699.555320:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828699.555323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333178496:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828699.938414:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828700.962348:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828701.986596:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828703.010699:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828704.034513:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828704.674343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674350:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828704.674354:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674361:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674371:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674377:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828704.674411:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333178624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.674435:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674439:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828704.674441:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674464:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828704.674477:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333178752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.674487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828704.674491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333178624:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828704.674492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674498:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674503:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828704.674518:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333178880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.674526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828704.674531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674538:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674541:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828704.674548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828704.674554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333179008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.674561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828704.674566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674572:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674575:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828704.674586:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb222a00 x1874663333178624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.674587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333179136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.674597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828704.674602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674606:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00080000:3.0:1787828704.674609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828704.674613:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333178624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.674613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828704.674623:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828704.674626:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333179264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828704.674628:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674632:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333178624:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828704.674635:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828704.674639:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00100000:2.0:1787828704.674654:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828704.674660:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674664:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828704.674668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828704.674674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828704.674679:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333179392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828704.674687:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb220700 x1874663333178752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.674689:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.674693:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828704.674694:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.674698:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10515 00000100:00100000:2.0:1787828704.674700:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333178752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.674701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.674705:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828704.674708:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828704.674709:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828704.674711:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674715:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828704.674718:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333179520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828704.674721:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828704.674738:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb221880 x1874663333178880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674762:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333178880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674771:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674774:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333178880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828704.674781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828704.674796:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674804:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb223100 x1874663333179008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333179008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674817:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674821:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828704.674824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828704.674834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb222d80 x1874663333179136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333179136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674853:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828704.674874:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828704.674886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb221c00 x1874663333179264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333179264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828704.674927:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828704.674937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb222300 x1874663333179392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333179392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674956:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674958:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828704.674961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828704.674971:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828704.674976:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff9057fb220380 x1874663333179520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828704.674983:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333179520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828704.674992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828704.674994:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828704.674996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333179520:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828705.058329:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828706.082445:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828707.106531:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828708.130426:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828709.154448:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828709.794314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828709.794329:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794341:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794353:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794360:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828709.794404:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333179648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828709.794444:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794453:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794462:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828709.794471:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333179776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794478:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794481:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828709.794482:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794484:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794486:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794488:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794492:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828709.794498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333179904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828709.794507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794511:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794513:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794516:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794520:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828709.794526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333180032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828709.794537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794541:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794544:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794546:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828709.794555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333180160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828709.794564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794567:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794569:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794573:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828709.794584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333179648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828709.794588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333180288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828709.794609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794614:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794621:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828709.794633:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333180416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828709.794641:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828709.794643:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828709.794644:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828709.794647:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10520 00000100:00080000:0.0:1787828709.794649:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828709.794652:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828709.794656:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828709.794665:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333180544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828709.794710:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828709.794742:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828709.794762:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828709.794776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828709.794798:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828709.794811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828709.794840:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180544:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828710.178517:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828711.202433:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828712.226421:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828712.866529:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1787828712.866572:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828712.866590:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866629:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866639:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866647:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866656:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866666:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866678:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828712.866687:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828712.866693:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158c000 x1874663333180288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866715:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333180288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866724:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866729:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866733:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828712.866739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158f800 x1874663333180416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333180416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866752:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866754:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866756:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828712.866768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158e680 x1874663333180544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333180544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866783:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333180544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828712.866794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158df80 x1874663333179648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866823:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333179648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866836:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866839:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333179648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828712.866846:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158ca80 x1874663333179776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866856:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333179776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828712.866879:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158f100 x1874663333179904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333179904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333179904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828712.866915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158e300 x1874663333180032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866927:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333180032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866935:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828712.866948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828712] req@ffff9057f158dc00 x1874663333180160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.866965:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333180160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.866974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.866978:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828712.866981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180160:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787828713.250377:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828714.274379:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828714.914358:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914369:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828714.914373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914383:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914393:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914399:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828714.914432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333180672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828714.914465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828714.914471:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914480:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914493:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828714.914508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828714.914524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333180800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828714.914532:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333180672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828714.914541:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914544:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828714.914546:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914549:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914552:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914556:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828714.914560:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828714.914571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333180928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828714.914582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914586:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828714.914588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914592:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914600:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828714.914605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828714.914609:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828714.914617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828714.914630:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333181056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828714.914641:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828714.914646:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914650:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914654:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914659:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828714.914665:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb220380 x1874663333180672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828714.914667:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828714.914685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333181184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828714.914693:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333180672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828714.914699:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828714.914706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828714.914709:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828714.914710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828714.914721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828714.914722:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914730:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828714.914734:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333180672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828714.914739:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828714.914747:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828714.914758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333181312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828714.914768:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828714.914769:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180800:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828714.914773:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828714.914778:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914782:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914802:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914807:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828714.914813:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828714.914821:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828714.914828:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333181440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828714.914840:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828714.914841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb222300 x1874663333180800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828714.914846:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828714.914850:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828714.914859:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10525 00000100:00080000:3.0:1787828714.914866:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828714.914870:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828714.914874:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333180800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828714.914877:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828714.914892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828714.914894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333181568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828714.914900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.914910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828714.914922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828714.914947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.914960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb221c00 x1874663333180928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.914979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333180928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.914987:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.914991:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.914994:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333180928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828714.915001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828714.915022:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.915033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb222d80 x1874663333181056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.915047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333181056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.915054:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.915058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.915061:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828714.915066:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828714.915085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.915103:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb223100 x1874663333181184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.915117:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333181184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.915125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.915128:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.915131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828714.915151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828714.915171:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.915183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb221880 x1874663333181312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.915193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333181312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.915201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.915204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.915207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828714.915214:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828714.915266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.915278:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb220700 x1874663333181440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.915290:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333181440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.915298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.915302:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.915306:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828714.915313:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828714.915333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828714.915345:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff9057fb222a00 x1874663333181568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828714.915357:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333181568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828714.915364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828714.915367:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828714.915371:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333181568:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828715.298387:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828716.322404:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828717.346488:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828718.370401:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828719.394369:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828720.034493:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.034508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828720.034514:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.034530:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.034545:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828720.034554:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828720.034612:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333181696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828720.034646:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.034650:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828720.034653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.034658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.034664:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828720.034669:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828720.034681:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828720.034698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333181696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828720.034702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333181824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828720.034731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.034736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828720.034740:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.034747:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.034763:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828720.034769:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828720.034786:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828720.034818:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333181952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828720.034828:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.034831:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828720.034833:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.034836:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.034841:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828720.034850:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828720.034851:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828720.034863:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828720.034874:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333182080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828720.034888:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.034891:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828720.034894:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.034898:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.034901:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828720.034907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb222680 x1874663333181696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828720.034926:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828720.034944:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828720.034951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333181696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828720.034962:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333182208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828720.034972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.034982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.034990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333181696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828720.035010:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.035016:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828720.035030:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.035037:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.035043:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828720.035047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181824:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828720.035055:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828720.035063:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828720.035110:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333182336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828720.035112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035131:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb223800 x1874663333181824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828720.035131:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.035168:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828720.035171:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828720.035175:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333181824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828720.035177:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.035190:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828720.035198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035218:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828720.035220:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035222:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181824:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828720.035233:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828720.035241:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828720.035252:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333182464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828720.035263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828720.035267:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828720.035288:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828720.035292:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828720.035299:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10530 00000100:00080000:3.0:1787828720.035306:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828720.035312:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035350:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb221500 x1874663333181952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035371:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333181952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828720.035379:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828720.035381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035419:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035436:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333181952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828720.035437:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333182592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828720.035490:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.035513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035528:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb221180 x1874663333182080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035537:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333182080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.035553:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035555:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035557:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828720.035580:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.035596:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb220000 x1874663333182208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035613:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333182208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.035625:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035628:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035631:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828720.035653:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828720.035678:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035709:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb223b80 x1874663333182336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035724:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333182336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.035734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035739:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035743:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828720.035770:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828720.035819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035832:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb220a80 x1874663333182464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035844:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333182464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.035851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035855:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035859:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828720.035865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.035885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828720.035896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9057fb220e00 x1874663333182592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.035909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333182592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.035918:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.035922:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828720.035925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333182592:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828720.418422:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828721.442370:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828722.466439:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828723.490394:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828724.514513:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828725.154297:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154306:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828725.154309:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154318:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154327:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154333:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828725.154367:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333182720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154390:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154409:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.154411:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154414:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154417:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154419:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154426:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828725.154435:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333182848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154442:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154444:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.154445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154447:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154450:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154453:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828725.154459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333182976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154465:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.154469:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154472:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154474:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154476:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828725.154491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333182720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828725.154491:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333183104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154505:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.154508:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154511:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154513:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154515:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828725.154525:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333183232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154531:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154534:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.154535:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154538:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154575:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154578:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154581:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828725.154588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828725.154592:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333183360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154600:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.154605:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.154608:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154611:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154613:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154616:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828725.154624:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333183488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.154630:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.154633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.154635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828725.154637:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb220e00 x1874663333182720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828725.154641:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10535 00000100:00080000:3.0:1787828725.154647:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.154654:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828725.154659:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828725.154666:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333182720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828725.154671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333183616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828725.154692:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154707:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333182720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828725.154730:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.154770:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154781:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb220a80 x1874663333182848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154789:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333182848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.154803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.154815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154821:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb223b80 x1874663333182976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333182976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333182976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.154854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.154866:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb220000 x1874663333183104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333183104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154889:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828725.154895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.154905:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb221180 x1874663333183232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333183232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828725.154929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.154938:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb221500 x1874663333183360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333183360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154964:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.154967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.154976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.154982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb223800 x1874663333183488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.154988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333183488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.154993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.154994:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.154996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.155016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.155029:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828725.155035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9057fb222680 x1874663333183616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.155043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333183616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.155049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.155051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828725.155054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333183616:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828725.538391:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828726.562403:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828727.586403:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828728.610485:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828729.634407:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828730.274330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274341:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828730.274346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274356:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:3.0:1787828730.274366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828730.274374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828730.274407:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333183744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828730.274436:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828730.274440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274443:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:3.0:1787828730.274446:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828730.274448:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828730.274464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333183872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828730.274472:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828730.274478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333183744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828730.274479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:3.0:1787828730.274489:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828730.274491:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274494:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828730.274504:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333184000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828730.274510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828730.274516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274520:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:3.0:1787828730.274534:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828730.274538:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828730.274543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828730.274567:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333184128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828730.274575:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb222a00 x1874663333183744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828730.274584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828730.274592:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828730.274593:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333183744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828730.274603:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:2.0:1787828730.274608:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828730.274610:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828730.274614:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828730.274620:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828730.274621:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333183744:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828730.274631:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828730.274646:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828730.274650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333184256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828730.274664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828730.274667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828730.274672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828730.274675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274693:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00100000:2.0:1787828730.274696:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb220700 x1874663333183872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828730.274698:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828730.274706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828730.274716:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333183872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.274725:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828730.274728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333184384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828730.274731:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.274739:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333183872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828730.274752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184000:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828730.274757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828730.274762:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828730.274763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828730.274767:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00080000:3.0:1787828730.274774:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828730.274779:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828730.274782:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274787:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828730.274793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb221880 x1874663333184000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828730.274798:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333184512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828730.274810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333184000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828730.274818:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828730.274820:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.274827:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828730.274827:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828730.274833:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828730.274834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184000:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828730.274845:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10541 00000100:00100000:2.0:1787828730.274846:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184128:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828730.274853:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828730.274860:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828730.274867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828730.274867:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828730.274877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb223100 x1874663333184128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828730.274886:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333184640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828730.274889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333184128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.274911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.274914:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.274917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828730.274932:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828730.274944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828730.274950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb222d80 x1874663333184256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828730.274962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333184256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.274967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.274969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.274970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828730.274974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828730.274984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828730.274990:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb221c00 x1874663333184384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828730.274997:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333184384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.275001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.275003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.275005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828730.275008:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828730.275018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828730.275023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb222300 x1874663333184512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828730.275030:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333184512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.275038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.275040:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.275042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828730.275046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828730.275055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828730.275060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828730/real 1787828730] req@ffff9057fb220380 x1874663333184640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828730.275067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333184640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828730.275080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828730.275083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828730.275086:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333184640:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828730.658403:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828731.682401:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828732.706369:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828733.730436:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828734.754421:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828735.394297:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394306:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828735.394310:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394320:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394339:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828735.394349:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828735.394385:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333184768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828735.394409:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394412:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828735.394414:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394417:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394419:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828735.394422:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394429:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828735.394439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333184896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828735.394448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828735.394452:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394455:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828735.394461:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333184768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828735.394463:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828735.394478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333185024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828735.394489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828735.394494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394497:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394500:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828735.394502:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394506:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828735.394515:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333185152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828735.394522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828735.394526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394529:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10546 00000800:00000100:2.0:1787828735.394534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828735.394536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828735.394541:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828735.394555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333185280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828735.394561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828735.394566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394569:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394573:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828735.394576:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb220380 x1874663333184768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828735.394579:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828735.394600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333185408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828735.394603:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333184768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828735.394624:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394627:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828735.394628:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828735.394630:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828735.394637:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828735.394638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394659:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828735.394660:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333184768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828735.394668:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828735.394684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333185536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828735.394689:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184896:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828735.394700:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828735.394703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828735.394705:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828735.394710:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10546 00000100:00080000:3.0:1787828735.394716:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828735.394721:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828735.394721:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828735.394730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828735.394740:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb222300 x1874663333184896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828735.394743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333185664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828735.394772:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333184896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.394784:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.394787:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333184896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828735.394793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828735.394806:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.394814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb221c00 x1874663333185024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.394822:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333185024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.394830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.394832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828735.394836:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828735.394848:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.394854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb222d80 x1874663333185152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.394862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333185152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394868:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.394870:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.394872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828735.394876:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828735.394886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.394893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb223100 x1874663333185280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.394901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333185280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.394908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.394910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828735.394914:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828735.394925:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.394931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb221880 x1874663333185408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.394944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333185408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.394952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.394954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828735.394967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828735.394979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.394985:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb220700 x1874663333185536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.394993:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333185536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.394998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.395000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.395002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828735.395006:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828735.395016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828735.395022:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828735/real 1787828735] req@ffff9057fb222a00 x1874663333185664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828735.395047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333185664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828735.395058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828735.395060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828735.395063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333185664:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828735.778364:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828736.802382:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828737.826348:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828738.850656:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828739.874395:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828740.514384:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514395:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828740.514399:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514410:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514423:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514431:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828740.514478:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333185792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514527:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828740.514532:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514537:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514540:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514543:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828740.514552:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828740.514564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333185920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514590:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828740.514591:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514593:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514595:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514598:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828740.514602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828740.514622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333186048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828740.514634:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514637:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514640:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514644:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828740.514647:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333185792:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787828740.514649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828740.514663:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333186176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514671:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514674:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828740.514675:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514678:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514682:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828740.514685:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828740.514698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333186304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514712:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828740.514715:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514720:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514724:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514729:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828740.514735:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828740.514741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828740.514752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333186432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828740.514764:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828740.514765:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828740.514768:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:1.0:1787828740.514772:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828740.514776:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828740.514780:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828740.514789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78a9f80 x1874663333185792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828740.514793:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333186560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828740.514813:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828740.514815:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333185792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828740.514822:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828740.514828:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828740.514832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828740.514837:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10551 00000100:00080000:2.0:1787828740.514839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828740.514845:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828740.514846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333185792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828740.514853:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828740.514857:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185920:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787828740.514861:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828740.514877:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333186688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828740.514883:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.514899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78a8380 x1874663333185920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.514909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333185920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.514915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.514918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.514922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333185920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828740.514926:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828740.514937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.514943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78ab800 x1874663333186048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.514951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333186048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.514956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.514958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.514959:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828740.514980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828740.514991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.514998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78a9880 x1874663333186176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.515005:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333186176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.515010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.515012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.515014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828740.515017:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828740.515026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.515033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78aad80 x1874663333186304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.515040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333186304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.515045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.515047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.515049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828740.515052:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828740.515070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.515076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78a8000 x1874663333186432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.515083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333186432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.515087:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.515089:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.515091:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828740.515095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828740.515105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.515111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78abb80 x1874663333186560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.515119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333186560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.515124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.515126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.515129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828740.515132:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828740.515142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828740.515147:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828740/real 1787828740] req@ffff9057c78ab480 x1874663333186688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828740.515153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333186688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828740.515158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828740.515160:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828740.515162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333186688:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828740.898391:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828741.922466:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828742.946517:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828743.970471:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828744.994425:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828745.634390:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634399:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828745.634404:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634413:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634424:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634431:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828745.634468:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333186816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828745.634501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634505:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634507:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634510:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828745.634528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333186944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634537:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634540:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828745.634542:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634545:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634547:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634551:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634554:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828745.634563:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333187072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634600:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828745.634602:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634606:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634610:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634631:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634640:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828745.634657:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333187200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828745.634674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333186816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828745.634679:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634692:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828745.634698:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634706:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634711:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634715:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828745.634733:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333187328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634745:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634750:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828745.634770:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634774:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634778:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634782:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634786:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828745.634795:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333187456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634803:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634805:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828745.634807:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634810:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634822:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634826:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634830:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828745.634845:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828745.634845:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333187584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828745.634861:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828745.634864:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828745.634866:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828745.634870:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10556 00000100:00080000:3.0:1787828745.634874:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828745.634879:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828745.634886:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828745.634894:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333187712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828745.634940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb222680 x1874663333186816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.634974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333186816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.634993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635002:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635009:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333186816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828745.635023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828745.635059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635079:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb223800 x1874663333186944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635098:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333186944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635117:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635123:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333186944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828745.635133:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828745.635165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb221500 x1874663333187072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635202:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333187072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635278:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635285:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828745.635330:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828745.635363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb221180 x1874663333187200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635402:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333187200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635421:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635426:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828745.635437:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828745.635470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635488:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb220000 x1874663333187328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635507:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333187328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635520:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635525:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635531:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828745.635541:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828745.635569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635611:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb223b80 x1874663333187456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635653:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333187456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828745.635686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828745.635716:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635733:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb220a80 x1874663333187584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635752:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333187584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635775:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635780:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828745.635796:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828745.635824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828745.635839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff9057fb220e00 x1874663333187712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828745.635857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333187712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828745.635880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828745.635885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828745.635890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333187712:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828746.018617:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828747.042377:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828748.066359:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828749.090511:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828750.114399:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828750.754305:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754313:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828750.754316:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828750.754333:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754345:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828750.754353:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828750.754391:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333187840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828750.754426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754431:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828750.754433:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828750.754437:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754442:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828750.754446:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828750.754456:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828750.754474:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333187968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828750.754486:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754490:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828750.754492:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828750.754496:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754500:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828750.754505:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828750.754510:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333187840:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828750.754514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828750.754533:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333188096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828750.754562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828750.754569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828750.754573:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754578:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828750.754582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828750.754587:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828750.754595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828750.754610:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333188224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828750.754620:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb220e00 x1874663333187840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754623:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828750.754636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828750.754643:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00100000:2.0:1787828750.754644:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333187840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754649:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828750.754658:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828750.754659:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828750.754664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.754668:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333187840:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828750.754669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828750.754677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828750.754689:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333188352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828750.754698:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828750.754708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828750.754709:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb220a80 x1874663333187968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754717:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828750.754722:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828750.754726:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333187968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754731:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754739:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828750.754744:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828750.754747:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828750.754751:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828750.754752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.754758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333187968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828750.754769:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333188480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828750.754780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828750.754784:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828750.754790:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828750.754793:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828750.754799:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828750.754800:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754806:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828750.754813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb223b80 x1874663333188096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754814:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828750.754825:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828750.754828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333188096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828750.754836:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.754838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828750.754843:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333188608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828750.754844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828750.754854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188224:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828750.754869:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828750.754875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828750.754876:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828750.754881:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828750.754885:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb220000 x1874663333188224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754889:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10561 00000100:00080000:3.0:1787828750.754899:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828750.754900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333188224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828750.754909:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828750.754911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.754918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.754921:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188224:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787828750.754922:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828750.754928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828750.754944:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333188736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828750.754945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828750.754955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb221180 x1874663333188352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828750.754967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333188352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828750.754972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.754974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.754983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828750.754994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828750.755005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828750.755012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb221500 x1874663333188480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828750.755019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333188480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828750.755024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.755026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.755028:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828750.755032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828750.755041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828750.755047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb223800 x1874663333188608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828750.755054:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333188608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828750.755059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.755061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.755063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828750.755066:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828750.755095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828750.755102:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828750] req@ffff9057fb222680 x1874663333188736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828750.755110:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333188736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828750.755116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828750.755118:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828750.755120:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333188736:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828751.138393:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828752.162472:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828753.186551:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828754.210427:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828755.234463:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828755.874356:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874364:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828755.874367:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874375:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874384:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874391:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828755.874423:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333188864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828755.874447:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874450:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828755.874452:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874457:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874460:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874465:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828755.874474:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828755.874486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333188992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828755.874493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333188864:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828755.874496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828755.874515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874518:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874523:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874527:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828755.874531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828755.874542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333189120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828755.874558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828755.874562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874568:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874572:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828755.874575:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828755.874577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828755.874585:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333189248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828755.874593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874595:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828755.874597:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874600:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874603:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874607:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874610:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb222a00 x1874663333188864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828755.874611:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828755.874620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333189376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828755.874638:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333188864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828755.874639:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874643:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828755.874645:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828755.874648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828755.874649:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874651:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.874653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828755.874655:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333188864:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828755.874659:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828755.874667:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333189504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828755.874675:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828755.874678:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188992:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828755.874678:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828755.874680:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874683:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874686:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874689:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828755.874694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828755.874698:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828755.874703:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333189632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828755.874710:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb220700 x1874663333188992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828755.874713:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828755.874717:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828755.874719:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828755.874722:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10566 00000100:00080000:3.0:1787828755.874726:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828755.874729:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874732:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333188992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828755.874734:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828755.874741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874744:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828755.874745:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333189760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828755.874748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333188992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828755.874754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828755.874773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb221880 x1874663333189120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.874796:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333189120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.874805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874808:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.874811:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828755.874815:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828755.874831:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb223100 x1874663333189248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.874846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333189248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.874851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.874855:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828755.874858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828755.874867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb222d80 x1874663333189376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.874879:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333189376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.874885:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874886:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.874888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333189376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828755.874891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828755.874901:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb221c00 x1874663333189504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.874918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333189504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.874923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874925:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.874926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828755.874949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828755.874959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.874965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb222300 x1874663333189632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.874971:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333189632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.874976:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.874978:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.874980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828755.874984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828755.874993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828755.875007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff9057fb220380 x1874663333189760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828755.875014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333189760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.875023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828755.875025:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828755.875027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333189760:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828756.258439:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828757.282510:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828758.306376:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828759.330557:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828760.354546:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828760.994277:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994288:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828760.994292:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994303:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:3.0:1787828760.994315:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994323:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828760.994368:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333189888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.994393:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994396:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828760.994398:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994401:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:3.0:1787828760.994404:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994407:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828760.994416:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828760.994426:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333190016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828760.994433:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333189888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828760.994438:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994441:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.994443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994446:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:3.0:1787828760.994449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994451:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828760.994455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828760.994476:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333190144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.994485:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828760.994490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994493:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:3.0:1787828760.994526:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994529:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828760.994534:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828760.994541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828760.994551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333190272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.994564:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994568:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.994570:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994575:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:3.0:1787828760.994581:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994586:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828760.994590:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb220380 x1874663333189888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828760.994594:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828760.994619:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333189888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828760.994620:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333190400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.994643:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994646:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828760.994649:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994659:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10571 00000100:00080000:2.0:1787828760.994660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828760.994666:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828760.994668:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828760.994677:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828760.994678:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333189888:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828760.994698:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828760.994715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333190528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828760.994724:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190016:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828760.994729:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.994735:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.994738:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994746:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10571 00000800:00000100:2.0:1787828760.994752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828760.994753:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994761:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828760.994767:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb222300 x1874663333190016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828760.994768:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828760.994783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333190016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828760.994789:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333190656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828760.994792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.994799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.994807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190016:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828760.994813:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828760.994816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190144:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828760.994821:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.994826:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.994835:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10571 00000800:00000100:2.0:1787828760.994841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828760.994841:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.994850:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828760.994854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb221c00 x1874663333190144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828760.994857:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828760.994869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333190144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.994875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828760.994879:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333190784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828760.994880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.994886:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828760.994904:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828760.994932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828760.994947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb222d80 x1874663333190272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828760.994961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333190272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.994970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.994973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.994977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828760.994984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828760.995007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828760.995026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb223100 x1874663333190400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828760.995039:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333190400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.995045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.995047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.995050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333190400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828760.995066:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828760.995081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828760.995088:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb221880 x1874663333190528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828760.995097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333190528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.995102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.995104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.995106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828760.995109:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828760.995131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828760.995136:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb220700 x1874663333190656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828760.995153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333190656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.995159:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.995161:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.995163:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828760.995167:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828760.995178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828760.995185:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff9057fb222a00 x1874663333190784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828760.995192:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333190784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828760.995197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828760.995199:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828760.995201:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333190784:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828761.378568:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828762.402417:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828763.426526:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828764.450426:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828765.474581:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828766.114408:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114419:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828766.114424:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114454:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114480:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114486:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828766.114517:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333190912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114567:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828766.114570:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114576:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114581:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114587:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.114616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333190912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828766.114618:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333191040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114636:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114639:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828766.114641:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114644:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114648:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114651:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114655:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828766.114684:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333191168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114691:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828766.114694:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114696:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114700:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114702:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114705:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828766.114712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828766.114715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333191296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114722:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114725:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828766.114726:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114728:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114731:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114734:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114737:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828766.114745:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333191424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114751:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114754:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828766.114755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828766.114758:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78aaa00 x1874663333190912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828766.114760:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114766:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114773:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114778:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.114784:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333190912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828766.114788:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333191552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.114805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.114830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828766.114838:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828766.114841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333190912:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828766.114845:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828766.114851:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114858:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114864:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114869:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114877:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.114888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828766.114895:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333191680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828766.114908:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828766.114928:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828766.114930:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828766.114933:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10576 00000100:00080000:1.0:1787828766.114938:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828766.114944:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828766.114944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787828766.114953:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828766.114965:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333191808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828766.114974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78aa680 x1874663333191040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.114986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333191040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828766.115021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828766.115035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115043:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78a9c00 x1874663333191168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333191168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828766.115081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828766.115095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115102:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78aa300 x1874663333191296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115124:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333191296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115137:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828766.115149:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828766.115171:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78a9180 x1874663333191424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115206:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333191424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333191424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828766.115219:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828766.115281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115296:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78a8700 x1874663333191552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115311:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333191552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115320:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115322:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828766.115357:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828766.115377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115387:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78a9500 x1874663333191680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115397:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333191680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115406:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115409:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828766.115414:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828766.115428:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828766.115476:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828766] req@ffff9057c78ab100 x1874663333191808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828766.115484:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333191808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828766.115494:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828766.115496:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828766.115498:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333191808:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828766.498327:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828767.522526:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828768.546400:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828769.570451:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828770.594370:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828771.234332:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234341:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828771.234345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234353:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234363:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234368:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828771.234414:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333191936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828771.234442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234445:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234448:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234451:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234459:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828771.234469:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333192064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234477:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234483:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828771.234483:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333191936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828771.234488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234525:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234533:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828771.234586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333192192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828771.234602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234605:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234608:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234611:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828771.234624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333192320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828771.234653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234655:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234658:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234661:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828771.234669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828771.234673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333192448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234680:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828771.234684:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234688:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234691:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234693:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828771.234703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333192576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828771.234709:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234713:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828771.234714:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158d880 x1874663333191936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828771.234716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828771.234722:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234743:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234752:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234758:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828771.234759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333191936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.234773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828771.234779:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333192704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828771.234779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.234788:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333191936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828771.234804:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828771.234808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828771.234811:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828771.234816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192064:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828771.234817:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10581 00000100:00080000:0.0:1787828771.234840:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828771.234845:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828771.234850:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828771.234862:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828771.234863:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333192832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828771.234883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158ce00 x1874663333192064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.234898:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333192064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.234912:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.234915:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.234918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.234922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828771.234938:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.234946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158c700 x1874663333192192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.234953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333192192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.234958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.234960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.234962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.234966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828771.234976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.234982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158d180 x1874663333192320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.234989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333192320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.234994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.234995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.234997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828771.235001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828771.235010:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.235016:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158d500 x1874663333192448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.235023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333192448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.235028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.235030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.235032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333192448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828771.235035:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828771.235045:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.235050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158ed80 x1874663333192576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.235057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333192576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.235062:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.235064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.235066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.235069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828771.235086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.235092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158fb80 x1874663333192704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.235099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333192704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.235104:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.235106:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.235109:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.235123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828771.235134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828771.235140:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828771/real 1787828771] req@ffff9057f158ea00 x1874663333192832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.235147:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333192832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.235152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.235154:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828771.235155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333192832:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828771.618466:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828772.642471:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828773.666402:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828774.690533:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828775.714457:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828776.354312:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354320:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828776.354324:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354332:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354340:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354346:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828776.354390:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333192960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828776.354411:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354413:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828776.354415:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354417:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354420:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354422:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354430:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828776.354439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333193088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828776.354448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333192960:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828776.354450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828776.354458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828776.354477:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333193216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828776.354483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828776.354487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354490:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354499:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354502:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354506:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828776.354513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333193344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828776.354519:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354522:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828776.354523:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354526:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354528:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354531:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354535:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828776.354542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333193472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828776.354546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193088:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828776.354557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828776.354562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354565:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354568:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354570:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828776.354584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828776.354593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333193600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828776.354601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828776.354607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828776.354608:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193344:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828776.354615:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354618:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828776.354621:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828776.354637:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828776.354638:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333193728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828776.354656:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828776.354658:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828776.354660:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828776.354664:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10587 00000100:00080000:3.0:1787828776.354667:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828776.354668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193600:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828776.354672:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828776.354678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828776.354688:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333193856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828776.354696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828776.354713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193856:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828776.738388:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828777.762421:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828778.786366:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828779.426508:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787828779.426545:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828779.426561:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426593:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426600:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426606:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426614:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426624:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426635:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828779.426646:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828779.426653:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb220000 x1874663333193472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426681:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333193472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828779.426710:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb223b80 x1874663333193600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426723:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333193600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426732:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426736:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828779.426747:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb223800 x1874663333193088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333193088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426769:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426773:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426777:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828779.426783:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb221500 x1874663333193216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333193216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426803:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426807:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426811:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828779.426817:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb221180 x1874663333193344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426830:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333193344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426839:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333193344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828779.426853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb222680 x1874663333192960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426867:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333192960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426879:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426883:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333192960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828779.426914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb220e00 x1874663333193856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426928:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333193856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828779.426959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828776/real 1787828779] req@ffff9057fb220a80 x1874663333193728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828779.426972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333193728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828779.426980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828779.426984:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828779.426987:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333193728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1787828779.810381:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828780.834416:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828781.474306:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474316:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828781.474320:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474346:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474355:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828781.474370:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828781.474410:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333193984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828781.474432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474435:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828781.474436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474439:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828781.474445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474451:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828781.474461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333194112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828781.474474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828781.474480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828781.474481:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333193984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828781.474485:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474487:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828781.474489:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474492:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828781.474499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333194240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828781.474509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828781.474514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474519:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474522:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828781.474526:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474532:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828781.474545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333194368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828781.474554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828781.474561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828781.474562:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828781.474569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474575:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828781.474580:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828781.474599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333194496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828781.474608:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb220a80 x1874663333193984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828781.474612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828781.474626:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474634:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474639:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828781.474643:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333193984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828781.474647:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474655:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828781.474662:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.474670:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828781.474672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333194624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828781.474678:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333193984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828781.474693:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828781.474696:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828781.474699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474703:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474707:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828781.474709:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194112:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828781.474714:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828781.474736:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333194752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828781.474747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828781.474762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828781.474767:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb220e00 x1874663333194112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828781.474768:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828781.474772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828781.474781:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10592 00000100:00080000:3.0:1787828781.474788:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828781.474792:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333194112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828781.474794:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828781.474804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828781.474808:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828781.474819:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333194880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828781.474822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.474831:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828781.474843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828781.474885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.474898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb222680 x1874663333194240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.474911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333194240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.474926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.474930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.474933:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828781.474940:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828781.474960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.474972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb221180 x1874663333194368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.474983:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333194368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.474991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.474994:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.474997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828781.475002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828781.475019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.475029:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb221500 x1874663333194496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.475039:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333194496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.475069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.475071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.475075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333194496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828781.475081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828781.475099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.475111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb223800 x1874663333194624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.475122:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333194624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.475129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.475132:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.475135:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828781.475153:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828781.475174:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.475185:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb223b80 x1874663333194752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.475197:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333194752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.475205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.475208:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.475211:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828781.475217:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828781.475273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828781.475290:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828781/real 1787828781] req@ffff9057fb220000 x1874663333194880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828781.475302:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333194880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828781.475312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828781.475315:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828781.475319:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333194880:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828781.858472:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828782.882411:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828783.906445:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828784.930393:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828785.954371:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828786.594358:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594367:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828786.594370:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594379:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594394:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594400:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828786.594433:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333195008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.594460:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594463:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828786.594465:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594470:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594475:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594480:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828786.594490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828786.594507:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333195136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828786.594518:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333195008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828786.594523:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594535:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.594537:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594541:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594545:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594551:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828786.594556:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828786.594571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333195264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.594583:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828786.594590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594593:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594597:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594599:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828786.594604:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828786.594614:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333195392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828786.594625:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828786.594627:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594631:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.594632:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594636:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594638:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594642:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828786.594645:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828786.594656:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333195520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.594663:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828786.594668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594671:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594675:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594679:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828786.594688:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78a8a80 x1874663333195008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828786.594690:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828786.594711:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333195648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828786.594719:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333195008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828786.594724:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.594743:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594750:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594757:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594760:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828786.594763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828786.594764:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828786.594779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828786.594780:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333195776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828786.594794:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333195008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828786.594796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.594813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.594815:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.594818:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10597 00000100:00080000:1.0:1787828786.594821:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.594824:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828786.594829:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828786.594833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828786.594840:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333195904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828786.594868:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.594882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78ab100 x1874663333195136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.594897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333195136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.594904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.594906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.594909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828786.594913:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828786.594926:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.594932:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78a9500 x1874663333195264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.594940:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333195264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.594945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.594948:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.594949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828786.594953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828786.594976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.594983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78a8700 x1874663333195392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.594991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333195392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.595001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.595003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.595006:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828786.595009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828786.595020:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.595027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78a9180 x1874663333195520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.595034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333195520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.595039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.595041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.595043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333195520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828786.595047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828786.595058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.595065:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78aa300 x1874663333195648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.595072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333195648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.595083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.595085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.595088:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828786.595092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828786.595104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.595111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78a9c00 x1874663333195776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.595118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333195776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.595124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.595126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.595128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828786.595142:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828786.595153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828786.595160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff9057c78aa680 x1874663333195904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828786.595167:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333195904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828786.595172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828786.595174:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828786.595176:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333195904:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828786.978447:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828788.002392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828789.026529:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828790.050483:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828791.074403:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828791.714334:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828791.714345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714353:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714363:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714369:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828791.714401:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333196032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714424:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828791.714428:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714431:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714433:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714436:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828791.714444:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828791.714453:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333196160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714461:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714463:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828791.714466:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828791.714468:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333196032:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828791.714471:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714478:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714480:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828791.714483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828791.714493:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333196288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714499:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714502:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828791.714504:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714506:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714523:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714526:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828791.714530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828791.714534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828791.714542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333196416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714551:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828791.714555:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714558:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714561:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714564:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828791.714568:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828791.714572:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb222a00 x1874663333196032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828791.714578:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333196544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714590:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828791.714594:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333196032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828791.714595:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828791.714598:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714606:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714611:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828791.714613:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828791.714615:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828791.714620:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828791.714620:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828791.714627:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333196032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828791.714632:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333196672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828791.714640:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714643:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828791.714644:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00080000:3.0:1787828791.714649:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828791.714652:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196160:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828791.714654:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828791.714661:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828791.714672:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333196800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828791.714680:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828791.714682:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828791.714685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828791.714687:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828791.714691:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10602 00000100:00100000:2.0:1787828791.714695:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb220700 x1874663333196160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828791.714696:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828791.714704:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828791.714714:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333196160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828791.714718:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828791.714726:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.714730:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828791.714733:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333196928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828791.714736:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828791.714748:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828791.714772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.714780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb221880 x1874663333196288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.714792:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333196288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.714803:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.714805:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.714807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828791.714812:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828791.714867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.714874:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb223100 x1874663333196416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.714882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333196416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.714887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.714890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.714892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828791.714896:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828791.714907:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.714913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb222d80 x1874663333196544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.714920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333196544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.714935:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.714938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.714940:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333196544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828791.714943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828791.714955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.714961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb221c00 x1874663333196672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.714969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333196672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.714974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.714975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.714977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828791.714990:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828791.715001:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.715007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb222300 x1874663333196800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.715014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333196800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.715019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.715021:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.715023:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828791.715027:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828791.715037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828791.715049:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff9057fb220380 x1874663333196928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828791.715056:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333196928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828791.715061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828791.715063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828791.715065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333196928:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828792.098430:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828793.122403:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828794.146387:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828795.170391:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828796.194475:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828796.834323:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834331:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828796.834335:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834343:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834357:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828796.834391:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333197056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834414:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834417:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828796.834418:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834421:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834424:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834427:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834434:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.834443:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333197184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.834455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834457:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834459:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834467:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.834474:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333197312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834482:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834484:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828796.834485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834491:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834497:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828796.834499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333197056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828796.834502:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834518:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828796.834527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333197440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834534:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834537:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.834538:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834546:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828796.834560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333197568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834582:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834584:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828796.834585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834588:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834591:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834594:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.834605:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333197696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834612:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834616:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.834618:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834621:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834624:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834628:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834634:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.834646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333197824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.834657:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.834661:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.834663:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.834667:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10607 00000100:00080000:3.0:1787828796.834671:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.834675:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828796.834679:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828796.834688:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333197952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828796.835747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.835811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb220380 x1874663333197056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.835828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333197056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.835838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.835841:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.835844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333197056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828796.835851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828796.835872:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.835881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb222300 x1874663333197184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.835889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333197184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.835895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.835898:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.835900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828796.835923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828796.835935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.835942:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb221c00 x1874663333197312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.835949:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333197312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.835955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.835958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.835960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828796.835964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828796.835975:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.835981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb222d80 x1874663333197440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.835994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333197440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.836015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.836017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.836019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828796.836023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828796.836035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.836041:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb223100 x1874663333197568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.836048:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333197568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.836053:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.836055:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.836057:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333197568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828796.836061:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828796.836090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.836097:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb221880 x1874663333197696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.836104:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333197696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.836114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.836116:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.836118:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828796.836122:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828796.836132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.836138:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb220700 x1874663333197824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.836144:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333197824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.836149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.836151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.836153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828796.836156:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828796.836167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828796.836173:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff9057fb222a00 x1874663333197952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828796.836179:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333197952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828796.836184:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828796.836186:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828796.836188:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333197952:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828797.218413:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828798.242437:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828799.266376:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828800.290480:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828801.314354:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828801.954316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.954330:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828801.954336:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954364:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954374:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828801.954425:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333198080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.954465:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.954471:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828801.954474:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954481:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954488:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954496:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828801.954538:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333198080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828801.954542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333198208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.954553:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.954555:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.954556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954559:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954562:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954565:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828801.954602:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333198336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.954615:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828801.954626:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828801.954633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828801.954639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954648:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954670:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954676:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954682:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828801.954687:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb220000 x1874663333198080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828801.954701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333198464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828801.954705:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333198080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.954728:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.954733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828801.954734:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828801.954738:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333198080:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828801.954740:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.954745:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954750:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954753:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954756:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954760:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828801.954763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828801.954773:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333198592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.954782:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828801.954786:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828801.954788:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828801.954791:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954796:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954799:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828801.954800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb223b80 x1874663333198208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828801.954805:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954812:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828801.954818:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333198208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828801.954823:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333198720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828801.954832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.954840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828801.954840:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828801.954847:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198208:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828801.954847:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.954850:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954857:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00100000:2.0:1787828801.954858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198336:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828801.954861:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.954868:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954873:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828801.954883:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333198848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828801.954887:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828801.954897:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828801.954900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb223800 x1874663333198336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828801.954901:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.954904:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.954911:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10612 00000100:00080000:3.0:1787828801.954917:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828801.954922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333198336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828801.954924:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828801.954932:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828801.954934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.954938:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.954941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828801.954944:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333198976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828801.954950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828801.954970:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828801.954978:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb221500 x1874663333198464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828801.954985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333198464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.954990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.954992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.954994:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828801.955003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828801.955014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828801.955020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb221180 x1874663333198592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828801.955031:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333198592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.955036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.955038:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.955040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333198592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828801.955043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828801.955052:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828801.955058:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb222680 x1874663333198720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828801.955065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333198720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.955069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.955071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.955073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828801.955076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828801.955085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828801.955090:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb220e00 x1874663333198848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828801.955097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333198848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.955106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.955108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.955110:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828801.955113:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828801.955122:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828801.955128:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff9057fb220a80 x1874663333198976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828801.955134:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333198976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828801.955139:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828801.955141:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828801.955143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333198976:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828802.338410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828803.362421:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828804.386430:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828805.410384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828806.434428:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828807.074336:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074347:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828807.074351:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074362:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074373:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074382:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828807.074461:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333199104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828807.074509:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074515:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828807.074517:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074523:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074527:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828807.074544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828807.074560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333199232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828807.074572:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828807.074578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074582:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074586:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074589:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828807.074594:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828807.074604:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333199104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828807.074619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333199360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828807.074647:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074650:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828807.074652:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074656:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074675:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074678:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828807.074686:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828807.074687:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828807.074703:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333199488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828807.074711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828807.074716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074722:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074726:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074731:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828807.074735:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828807.074736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb220a80 x1874663333199104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828807.074750:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333199616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828807.074759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333199104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828807.074762:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074775:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828807.074780:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828807.074795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828807.074795:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074802:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828807.074807:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828807.074813:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828807.074818:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333199104:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787828807.074820:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828807.074837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828807.074840:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333199744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828807.074852:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074854:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828807.074857:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828807.074860:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074863:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828807.074865:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828807.074869:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828807.074875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828807.074886:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333199872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828807.074892:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb220e00 x1874663333199232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828807.074921:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828807.074925:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828807.074926:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828807.074930:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333199232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828807.074944:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10617 00000100:00080000:3.0:1787828807.074954:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828807.074959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828807.074961:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828807.074968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828807.074969:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828807.074977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828807.074985:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333200000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828807.075009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828807.075027:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb222680 x1874663333199360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333199360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075051:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828807.075055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.075066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb221180 x1874663333199488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333199488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075084:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075086:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075088:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828807.075092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.075102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075108:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb221500 x1874663333199616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075115:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333199616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075120:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075122:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075124:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333199616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828807.075127:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333199744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828807.075137:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075143:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb223800 x1874663333199744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075171:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333199744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075178:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075180:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333199744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828807.075193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333199872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828807.075205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075212:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb223b80 x1874663333199872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075249:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333199872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075258:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075260:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333199872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828807.075265:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.075278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828807.075285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9057fb220000 x1874663333200000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.075293:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333200000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.075303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.075305:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828807.075307:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200000:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828807.458421:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828808.482428:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828809.506446:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828810.530565:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828811.554396:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828812.194313:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.194325:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828812.194331:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194359:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00080000:3.0:1787828812.194370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194379:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828812.194444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333200128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.194493:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.194498:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.194501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194508:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00080000:3.0:1787828812.194513:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194519:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.194532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828812.194551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333200256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.194564:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333200128:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828812.194573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.194580:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.194582:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194589:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00080000:3.0:1787828812.194594:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194600:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.194607:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828812.194625:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333200384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.194640:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.194645:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.194649:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194654:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10622 00000800:00000100:2.0:1787828812.194662:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828812.194665:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194674:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.194693:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828812.194712:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb222a00 x1874663333200128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828812.194716:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333200512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.194777:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828812.194783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333200128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828812.194803:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.194805:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194810:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00080000:3.0:1787828812.194816:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194826:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828812.194828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.194835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828812.194836:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828812.194842:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333200128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828812.194857:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333200640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.194868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200256:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.194876:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.194882:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.194885:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.194892:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10622 00000800:00000100:2.0:1787828812.194914:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828812.194917:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.194927:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.194936:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828812.194937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb220700 x1874663333200256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828812.194974:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333200768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.194985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333200256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828812.195052:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828812.195054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828812.195067:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828812.195069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200256:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.195074:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.195091:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00100000:2.0:1787828812.195093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.195103:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.195145:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.195172:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828812.195180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828812.195197:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333200896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.195200:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb221880 x1874663333200384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333200384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828812.195308:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828812.195319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828812.195321:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.195332:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828812.195333:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195364:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.195364:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10622 00000100:00080000:3.0:1787828812.195373:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828812.195377:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200512:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828812.195383:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828812.195391:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828812.195411:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333201024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828812.195413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828812.195460:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb223100 x1874663333200512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195492:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333200512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.195516:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195523:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828812.195557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.195594:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828812.195611:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb222d80 x1874663333200640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195626:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333200640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.195641:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195645:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333200640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828812.195668:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828812.195685:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828812.195694:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb221c00 x1874663333200768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195705:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333200768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.195714:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828812.195722:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828812.195735:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828812.195759:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb222300 x1874663333200896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195768:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333200896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195774:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.195776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333200896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828812.195782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.195794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828812.195801:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9057fb220380 x1874663333201024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.195809:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333201024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.195815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.195817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828812.195819:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201024:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828812.578571:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828813.602423:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828814.626416:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828815.650417:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828816.674500:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828817.314324:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314332:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828817.314335:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314343:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314352:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314358:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828817.314400:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333201152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314423:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828817.314429:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314437:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828817.314450:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828817.314464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333201280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314485:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828817.314487:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333201152:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828817.314488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828817.314490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314494:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314497:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314500:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828817.314505:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828817.314514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333201408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314522:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314525:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828817.314526:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314529:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314532:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314535:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828817.314556:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828817.314561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828817.314566:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333201536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314575:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314579:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828817.314580:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314584:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314588:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314592:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828817.314598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828817.314606:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78aaa00 x1874663333201152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828817.314608:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333201664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314621:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828817.314628:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828817.314629:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333201152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828817.314633:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314637:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314641:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828817.314642:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828817.314647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828817.314647:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.314652:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333201152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828817.314659:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333201792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314668:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828817.314672:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828817.314674:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828817.314676:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201280:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828817.314679:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00080000:1.0:1787828817.314699:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828817.314703:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828817.314709:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828817.314719:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828817.314722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333201920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828817.314735:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828817.314735:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78ab480 x1874663333201280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828817.314739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828817.314742:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828817.314747:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10628 00000100:00100000:2.0:1787828817.314751:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333201280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.314762:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828817.314764:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828817.314766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828817.314769:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828817.314771:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201280:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787828817.314775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828817.314779:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828817.314788:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333202048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828817.314802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.314816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78abb80 x1874663333201408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.314833:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333201408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.314844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.314849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.314854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828817.314860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828817.314881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.314891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78a8000 x1874663333201536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.314901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333201536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.314907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.314910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.314913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828817.314917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828817.314955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.314960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78aad80 x1874663333201664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.314967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333201664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.314972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.314974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.314976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333201664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828817.314979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828817.314989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.314995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78a9880 x1874663333201792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.315007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333201792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.315012:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.315013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.315015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828817.315027:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828817.315037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.315044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78ab800 x1874663333201920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.315051:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333201920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.315056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.315058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.315059:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333201920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828817.315062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828817.315072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828817.315097:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828817/real 1787828817] req@ffff9057c78a8380 x1874663333202048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828817.315116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333202048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828817.315125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828817.315127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828817.315129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202048:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828817.698391:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828818.722475:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828819.746391:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828820.770539:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828821.794373:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828822.434330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434339:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828822.434342:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434360:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434366:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828822.434402:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333202176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828822.434432:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828822.434444:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434451:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434478:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828822.434495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333202304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828822.434496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333202176:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828822.434514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828822.434536:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434541:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434546:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434551:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828822.434573:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333202432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828822.434579:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828822.434588:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434592:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828822.434595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434601:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434606:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434612:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434619:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828822.434629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78a9f80 x1874663333202176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828822.434635:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333202560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828822.434663:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828822.434669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828822.434675:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333202176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828822.434676:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434684:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828822.434691:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828822.434692:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828822.434698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787828822.434703:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828822.434704:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333202176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828822.434723:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333202688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828822.434733:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202304:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828822.434735:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434760:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828822.434762:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434767:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434772:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434777:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434783:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828822.434789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828822.434801:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333202816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828822.434827:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828822.434830:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828822.434831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828822.434837:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00100000:2.0:1787828822.434839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78a8380 x1874663333202304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828822.434840:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828822.434847:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434854:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828822.434864:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333202304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828822.434883:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333202944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828822.434901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828822.434904:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828822.434907:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828822.434909:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828822.434913:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828822.434917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202304:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828822.434920:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10633 00000100:00080000:1.0:1787828822.434926:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828822.434929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202432:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828822.434932:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828822.434954:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828822.434965:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333203072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828822.434973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.434986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78ab800 x1874663333202432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.434996:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333202432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435003:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435005:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828822.435011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828822.435023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.435030:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78a9880 x1874663333202560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.435037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333202560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435062:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828822.435076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828822.435091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.435100:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78aad80 x1874663333202688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.435108:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333202688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435115:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333202688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828822.435120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828822.435131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.435137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78a8000 x1874663333202816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.435145:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333202816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435152:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828822.435157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828822.435167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.435172:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78abb80 x1874663333202944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.435180:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333202944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435187:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435189:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333202944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828822.435193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828822.435203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828822.435208:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828822/real 1787828822] req@ffff9057c78ab480 x1874663333203072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828822.435215:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333203072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828822.435240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828822.435243:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828822.435245:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203072:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828822.818384:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828823.842391:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828824.866505:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828825.890384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828826.914474:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828827.554329:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828827.554345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554361:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554374:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828827.554426:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333203200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828827.554456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828827.554483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333203328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554494:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828827.554495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554498:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554500:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828827.554505:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333203200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828827.554505:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554515:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828827.554529:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333203456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554540:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828827.554548:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554552:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554570:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554575:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828827.554581:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828827.554595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333203584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554603:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828827.554607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554611:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554614:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554617:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828827.554625:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb220380 x1874663333203200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828827.554632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333203712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828827.554647:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333203200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828827.554647:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828827.554651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554658:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554664:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828827.554666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828827.554667:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828827.554675:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828827.554675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828827.554683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333203200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828827.554687:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333203840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828827.554693:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203328:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828827.554699:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554702:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828827.554704:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828827.554708:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554710:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554713:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554716:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828827.554721:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828827.554726:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333203968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828827.554734:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828827.554738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828827.554740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828827.554741:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb222300 x1874663333203328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828827.554745:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10638 00000100:00080000:3.0:1787828827.554748:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828827.554754:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828827.554761:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828827.554768:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333204096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828827.554783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333203328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.554792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.554795:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.554799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828827.554826:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828827.554843:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.554851:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb221c00 x1874663333203456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.554859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333203456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.554865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.554867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.554870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828827.554873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828827.554885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.554891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb222d80 x1874663333203584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.554899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333203584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.554904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.554906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.554908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828827.554912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828827.554922:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.554928:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb223100 x1874663333203712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.554935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333203712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.554941:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.554942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.554944:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333203712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828827.554947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828827.554958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.554963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb221880 x1874663333203840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.554976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333203840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.554981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.554983:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.554985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828827.554988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828827.554998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.555018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb220700 x1874663333203968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.555026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333203968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.555032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.555034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.555036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333203968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828827.555039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828827.555053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828827.555060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828827/real 1787828827] req@ffff9057fb222a00 x1874663333204096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828827.555067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333204096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828827.555077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828827.555079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828827.555081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204096:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828827.938467:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828828.962407:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828829.986439:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828831.010452:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828832.034605:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828832.674315:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674324:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828832.674328:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674337:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674347:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674353:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828832.674394:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333204224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674427:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674431:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828832.674433:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674437:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674441:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674453:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828832.674466:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333204352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674475:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828832.674480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333204224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828832.674480:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828832.674483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674490:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674494:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674496:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828832.674511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333204480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828832.674528:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674530:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674557:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674563:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828832.674564:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674569:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828832.674581:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333204608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674588:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828832.674592:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674595:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674599:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674602:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674622:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828832.674631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb220000 x1874663333204224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828832.674631:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333204736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674657:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828832.674659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674662:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674666:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828832.674672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333204224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828832.674674:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828832.674691:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828832.674692:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333204864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828832.674699:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828832.674706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828832.674709:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828832.674710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674713:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00100000:2.0:1787828832.674717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333204224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828832.674718:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674725:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828832.674735:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333204992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828832.674743:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828832.674745:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204352:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828832.674747:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828832.674752:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828832.674755:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10643 00000100:00080000:3.0:1787828832.674758:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828832.674761:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828832.674764:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828832.674772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828832.674776:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333205120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828832.674788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb223b80 x1874663333204352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.674799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333204352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.674806:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.674810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.674814:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.674818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828832.674848:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.674855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb223800 x1874663333204480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.674863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333204480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.674869:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.674871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.674873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.674877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828832.674889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.674912:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb221500 x1874663333204608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.674920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333204608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.674926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.674928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.674930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828832.674934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828832.674945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.674952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb221180 x1874663333204736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.674960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333204736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.674966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.674985:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.674988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333204736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828832.674992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828832.675008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.675016:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb222680 x1874663333204864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.675024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333204864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.675047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.675050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.675053:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.675069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828832.675083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.675091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb220e00 x1874663333204992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.675099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333204992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.675106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.675108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.675111:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333204992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.675115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333205120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828832.675126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828832.675133:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff9057fb220a80 x1874663333205120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.675142:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333205120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.675148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.675150:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828832.675152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333205120:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828833.058456:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828834.082376:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828835.106396:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828836.130358:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828837.154416:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828837.794349:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794360:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828837.794364:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794375:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794393:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794401:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828837.794447:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333205248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794480:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794483:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828837.794484:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794488:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794491:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794495:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794504:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828837.794514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333205376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794523:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794526:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.794527:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794529:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828837.794549:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333205248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828837.794558:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333205504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794567:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794569:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828837.794571:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794574:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794576:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794578:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828837.794590:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333205632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794596:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794599:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.794600:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794603:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794621:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794624:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794627:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828837.794637:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333205760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828837.794639:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828837.794649:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828837.794654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794657:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794662:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828837.794673:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333205888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794678:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.794681:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794684:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00080000:3.0:1787828837.794687:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794690:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828837.794694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828837.794708:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb220a80 x1874663333205248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828837.794708:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333206016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.794735:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.794740:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.794742:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.794749:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10648 00000100:00100000:2.0:1787828837.794750:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333205248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828837.794761:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.794768:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828837.794770:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828837.794775:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828837.794780:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.794787:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333205248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828837.794791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333206144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828837.794803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828837.794834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.794868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb220e00 x1874663333205376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.794884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333205376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.794894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.794898:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.794902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.794948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828837.794972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.794986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb222680 x1874663333205504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333205504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.795025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828837.795049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.795063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb221180 x1874663333205632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795078:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333205632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795087:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828837.795102:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828837.795124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.795137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb221500 x1874663333205760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795151:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333205760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795165:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795168:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333205760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828837.795175:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333205888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828837.795196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.795209:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb223800 x1874663333205888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795262:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333205888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795275:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795278:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333205888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.795286:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828837.795308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.795320:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb223b80 x1874663333206016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795333:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333206016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795345:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795349:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.795355:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828837.795375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828837.795397:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff9057fb220000 x1874663333206144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.795408:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333206144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.795424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.795427:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828837.795431:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206144:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828838.178406:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828839.202597:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828840.226476:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828841.250419:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828842.274368:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828842.914300:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914310:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828842.914314:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914324:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914334:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828842.914376:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333206272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914398:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914401:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828842.914402:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914405:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914407:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914410:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914417:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.914426:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333206400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914434:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828842.914438:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333206272:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828842.914442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914449:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914452:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914454:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.914464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333206528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914470:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828842.914474:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914476:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914478:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914480:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914484:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828842.914498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333206656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828842.914507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914509:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914512:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914514:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828842.914528:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333206784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914537:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828842.914539:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914541:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914545:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828842.914546:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206400:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828842.914550:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.914568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333206912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828842.914579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.914581:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914584:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914588:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828842.914588:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206528:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828842.914595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.914604:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333207040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.914612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.914614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828842.914615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828842.914621:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206656:192.168.203.156@tcp:8:kworker.0 00000100:00080000:0.0:1787828842.914622:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10653 00000100:00080000:0.0:1787828842.914628:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.914632:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828842.914635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828842.914646:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333207168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828842.914648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828842.914675:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828842.914696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828842.914718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207168:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828843.298437:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828844.322358:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828845.346476:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828845.986491:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787828845.986533:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828845.986549:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986585:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986594:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986604:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986612:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986624:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787828845.986634:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828845.986635:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158fb80 x1874663333206400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828845.986645:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828845.986669:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333206400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986690:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828845.986703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158ed80 x1874663333206528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986718:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333206528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986725:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986728:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828845.986736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158d500 x1874663333206656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986749:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333206656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828845.986772:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158d180 x1874663333206784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986784:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333206784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986796:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333206784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828845.986800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158ea00 x1874663333206272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986813:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333206272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333206272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828845.986856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158c700 x1874663333206912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333206912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333206912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828845.986905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158ce00 x1874663333207040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986919:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333207040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828845.986940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828845] req@ffff9057f158d880 x1874663333207168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828845.986953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333207168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828845.986960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828845.986963:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828845.986967:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207168:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787828846.370429:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828847.394448:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828848.034332:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828848.034340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828848.034344:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034352:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034360:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828848.034366:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828848.034394:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333207296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034420:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828848.034424:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828848.034426:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034431:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034435:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828848.034440:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828848.034450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.034462:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333207296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828848.034467:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333207424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828848.034486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828848.034488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034493:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828848.034502:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828848.034507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828848.034521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333207552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828848.034533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828848.034536:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828848.034541:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828848.034543:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034549:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034559:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828848.034564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828848.034571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828848.034577:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158dc00 x1874663333207296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.034587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333207680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828848.034605:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333207296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.034606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828848.034610:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034619:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.034631:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.034634:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828848.034640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:0.0:1787828848.034643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828848.034647:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333207296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828848.034661:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333207808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034676:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828848.034679:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207424:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828848.034684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828848.034689:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034694:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034698:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828848.034714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828848.034721:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828848.034728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.034734:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158e300 x1874663333207424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.034744:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333207936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828848.034757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333207424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.034767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828848.034770:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034779:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:2.0:1787828848.034780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.034784:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.034789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828848.034794:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828848.034799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207424:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828848.034802:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.034810:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787828848.034818:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333208064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.034836:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828848.034839:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828848.034841:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828848.034845:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.034851:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10658 00000100:00080000:0.0:1787828848.034855:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828848.034874:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828848.034875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158f100 x1874663333207552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828848.034879:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828848.034897:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333208192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828848.034902:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333207552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.034923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.034927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.034931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828848.034939:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828848.034962:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828848.034974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158ca80 x1874663333207680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.034984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333207680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.034992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.034995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.034998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828848.035015:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828848.035048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828848.035061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158df80 x1874663333207808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.035073:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333207808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.035082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.035085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.035088:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333207808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828848.035094:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828848.035113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828848.035124:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158e680 x1874663333207936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.035136:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333207936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.035144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.035147:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.035151:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333207936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828848.035157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828848.035178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828848.035190:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158f800 x1874663333208064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.035201:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333208064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.035209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.035212:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.035215:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828848.035253:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828848.035275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828848.035287:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9057f158c000 x1874663333208192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.035299:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333208192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.035310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.035313:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828848.035316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208192:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828848.418368:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828849.442431:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828850.466358:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828851.490478:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828852.514467:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828853.154374:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.154389:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828853.154394:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.154407:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.154422:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.154433:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828853.154497:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333208320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.154532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.154549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.154552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.154560:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.154567:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.154576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.154590:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828853.154626:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333208320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828853.154629:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333208448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.154655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.154662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.154665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.154672:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.154686:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.154693:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.154702:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828853.154725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333208576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828853.154736:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828853.154764:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.154770:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.154773:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.154782:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00100000:2.0:1787828853.154790:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb222a00 x1874663333208320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.154792:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.154806:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.154823:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828853.154871:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333208320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828853.154883:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333208704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828853.154903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828853.154908:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828853.154912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828853.154917:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828853.154925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333208320:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828853.154926:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.154939:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.154945:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.154950:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.154956:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828853.154967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828853.154976:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333208832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.154994:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.155000:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.155003:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828853.155010:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828853.155018:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.155027:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828853.155035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb220700 x1874663333208448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.155040:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.155071:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828853.155100:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333208960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828853.155109:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333208448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155167:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828853.155190:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828853.155197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208448:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828853.155203:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.155210:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828853.155211:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208576:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828853.155274:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.155282:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.155288:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828853.155298:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828853.155299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828853.155321:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333209088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828853.155324:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb221880 x1874663333208576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.155361:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.155373:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.155379:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828853.155389:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333208576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.155391:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10663 00000100:00080000:3.0:1787828853.155404:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828853.155406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828853.155416:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828853.155417:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155426:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208576:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787828853.155431:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828853.155447:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828853.155457:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333209216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828853.155474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828853.155492:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb223100 x1874663333208704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828853.155506:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333208704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155514:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155517:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155521:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828853.155539:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828853.155556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828853.155566:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb222d80 x1874663333208832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828853.155577:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333208832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155592:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155595:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155598:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333208832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828853.155603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828853.155621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828853.155642:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb221c00 x1874663333208960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828853.155660:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333208960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155673:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155679:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155684:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333208960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828853.155701:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828853.155730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828853.155743:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb222300 x1874663333209088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828853.155755:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333209088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155770:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828853.155776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828853.155798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828853.155814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9057fb220380 x1874663333209216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828853.155855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333209216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828853.155869:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828853.155874:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828853.155880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209216:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828853.538532:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828854.562427:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828855.586370:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828856.610346:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828857.634470:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828858.274295:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274304:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828858.274307:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274323:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274332:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274338:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828858.274373:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333209344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274395:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274398:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828858.274400:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274403:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274409:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274412:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274424:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828858.274435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333209472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274446:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828858.274448:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00100000:2.0:1787828858.274454:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333209344:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828858.274455:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274466:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828858.274474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333209600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274480:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274482:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828858.274484:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274486:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274503:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274506:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828858.274512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828858.274521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333209728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828858.274536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274542:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274545:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274548:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828858.274550:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb220380 x1874663333209344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828858.274558:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333209856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828858.274568:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333209344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828858.274569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828858.274573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274580:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274585:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828858.274586:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828858.274589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828858.274594:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828858.274596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828858.274600:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333209344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828858.274606:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333209984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828858.274613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828858.274616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828858.274621:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209472:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828858.274632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00080000:3.0:1787828858.274636:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274639:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828858.274651:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828858.274673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333210112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828858.274692:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb222300 x1874663333209472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828858.274697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828858.274708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828858.274712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828858.274722:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10669 00000100:00100000:2.0:1787828858.274728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333209472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828858.274730:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828858.274742:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828858.274769:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828858.274770:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.274776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.274779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828858.274782:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333210240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828858.274788:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828858.274853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.274868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb221c00 x1874663333209600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.274879:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333209600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.274887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.274890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.274892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828858.274897:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828858.274912:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.274920:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb222d80 x1874663333209728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.274936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333209728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.274943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.274945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.274948:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828858.274952:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828858.274965:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.274980:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb223100 x1874663333209856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.274990:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333209856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.274997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.274999:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.275002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333209856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828858.275006:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828858.275019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.275027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb221880 x1874663333209984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.275036:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333209984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.275049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.275052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.275055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333209984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828858.275072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333210112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828858.275086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.275094:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb220700 x1874663333210112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.275112:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333210112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.275119:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.275121:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.275124:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333210112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828858.275129:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333210240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828858.275142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828858.275149:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828858/real 1787828858] req@ffff9057fb222a00 x1874663333210240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828858.275159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333210240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828858.275166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828858.275168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828858.275171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333210240:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828858.658389:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828859.682388:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828860.706460:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828861.730405:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828862.754410:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828863.394391:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394402:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828863.394407:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394417:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394428:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394445:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828863.394503:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333210368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394540:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828863.394542:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394546:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394550:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394553:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394563:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828863.394586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333210496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394598:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394601:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828863.394606:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828863.394608:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333210368:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828863.394613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394625:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828863.394645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333210624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394658:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828863.394660:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394664:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394668:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394672:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394679:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828863.394692:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828863.394694:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333210752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394709:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828863.394711:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394717:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394720:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394723:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828863.394742:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333210880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394772:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828863.394776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78aaa00 x1874663333210368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828863.394778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828863.394782:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394794:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394802:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828863.394807:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394813:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828863.394814:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333210368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.394831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828863.394835:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333211008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828863.394840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.394852:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333210368:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828863.394871:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828863.394875:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828863.394877:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394881:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394885:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828863.394889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210496:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828863.394894:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394902:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828863.394915:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333211136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828863.394927:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828863.394929:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828863.394935:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828863.394939:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828863.394945:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10674 00000100:00080000:1.0:1787828863.394950:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828863.394954:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78aa680 x1874663333210496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828863.394959:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828863.394968:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828863.394978:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333210496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.394992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828863.394998:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333211264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828863.395001:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828863.395027:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828863.395053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395064:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78a9c00 x1874663333210624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333210624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395084:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395087:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828863.395092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828863.395122:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395132:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78aa300 x1874663333210752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395143:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333210752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395152:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828863.395161:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828863.395176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78a9180 x1874663333210880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395201:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333210880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395211:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395214:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333210880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828863.395218:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828863.395273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395284:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78a8700 x1874663333211008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395294:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333211008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395304:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395307:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828863.395324:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828863.395340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395349:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78a9500 x1874663333211136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333211136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395385:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395390:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828863.395399:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828863.395428:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828863.395445:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828863/real 1787828863] req@ffff9057c78ab100 x1874663333211264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828863.395462:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333211264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828863.395471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828863.395473:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828863.395477:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333211264:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828863.778406:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828864.802419:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828865.826393:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828866.850454:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828867.874449:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828868.514303:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514314:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828868.514318:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514328:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514339:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514346:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828868.514393:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333211392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514420:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514422:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828868.514424:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514427:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514430:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514440:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828868.514452:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333211520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514460:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514462:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828868.514463:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514465:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514468:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514474:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828868.514478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333211392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828868.514484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333211648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514493:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828868.514496:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514499:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514501:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514503:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514513:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828868.514522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333211776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828868.514531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514536:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10679 00000800:00000100:2.0:1787828868.514537:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828868.514539:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514543:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514547:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828868.514556:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333211904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514562:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514564:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828868.514566:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828868.514569:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514572:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828868.514575:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78a8a80 x1874663333211392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828868.514585:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514591:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828868.514596:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333211392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828868.514602:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333212032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828868.514624:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828868.514626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828868.514632:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828868.514635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828868.514639:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333211392:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828868.514642:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514645:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828868.514647:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828868.514650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828868.514661:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828868.514662:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333212160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828868.514676:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828868.514680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828868.514682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828868.514687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828868.514693:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10679 00000100:00080000:1.0:1787828868.514698:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828868.514700:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78ab100 x1874663333211520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828868.514707:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333211520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828868.514715:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828868.514737:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514739:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514743:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828868.514748:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333212288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828868.514778:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828868.514795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78a9500 x1874663333211648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333211648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514818:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828868.514823:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828868.514834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78a8700 x1874663333211776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333211776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514853:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828868.514858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828868.514867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78a9180 x1874663333211904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514879:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333211904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514886:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333211904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828868.514891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828868.514900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78aa300 x1874663333212032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333212032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828868.514928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828868.514937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514942:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78a9c00 x1874663333212160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514949:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333212160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514954:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514955:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828868.514960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828868.514969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828868.514974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828868/real 1787828868] req@ffff9057c78aa680 x1874663333212288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828868.514981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333212288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828868.514990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828868.514992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828868.514994:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333212288:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828868.898374:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828869.922358:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828870.946452:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828871.970469:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828872.994487:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828873.634337:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634345:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828873.634348:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634356:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634365:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634370:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828873.634403:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333212416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634429:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828873.634430:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634435:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634439:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634446:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828873.634460:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333212416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828873.634460:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333212544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634475:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634477:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828873.634479:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634482:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634485:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634487:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828873.634498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333212672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634503:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634505:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828873.634507:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634510:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634513:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634515:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828873.634523:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828873.634530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333212800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828873.634540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634545:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828873.634555:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333212928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634560:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634563:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828873.634564:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634566:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634568:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634571:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634574:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828873.634581:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333213056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634586:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634589:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828873.634590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634593:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634595:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634597:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828873.634601:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828873.634603:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de0a80 x1874663333212416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828873.634610:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333213184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828873.634620:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828873.634624:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828873.634625:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333212416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828873.634626:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828873.634635:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10684 00000100:00080000:2.0:1787828873.634640:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.634644:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828873.634645:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828873.634647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828873.634650:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634656:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333212416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828873.634658:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333213312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828873.634677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828873.634713:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634722:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de0000 x1874663333212544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634739:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333212544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634747:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634750:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634752:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828873.634756:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828873.634768:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634775:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de1c00 x1874663333212672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333212672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634795:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828873.634799:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828873.634810:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de1500 x1874663333212800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333212800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828873.634835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828873.634845:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de0e00 x1874663333212928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333212928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333212928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828873.634873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828873.634884:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634889:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de3100 x1874663333213056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333213056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634904:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828873.634907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828873.634918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de1f80 x1874663333213184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333213184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828873.634942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828873.634952:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828873.634957:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff9057d0de3800 x1874663333213312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828873.634964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333213312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828873.634969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828873.634971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828873.634973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333213312:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828874.018450:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828875.042582:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828876.066489:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828877.090470:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828878.114562:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828878.754378:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754387:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828878.754390:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754399:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754409:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754415:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828878.754451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333213440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828878.754479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754489:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828878.754511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333213568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754524:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828878.754526:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754528:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754530:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754533:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828878.754546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333213696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828878.754560:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333213440:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828878.754560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828878.754582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754586:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754589:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754593:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754598:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828878.754610:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333213824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754617:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754620:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828878.754621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754625:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754628:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828878.754644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333213952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754669:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828878.754671:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754675:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754678:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828878.754685:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828878.754687:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828878.754710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333214080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754719:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828878.754724:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754728:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754732:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754738:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754743:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828878.754750:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb220000 x1874663333213440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828878.754755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333214208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828878.754774:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828878.754795:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828878.754803:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828878.754820:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10689 00000100:00080000:3.0:1787828878.754823:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828878.754827:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828878.754832:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828878.754838:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333213440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828878.754847:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333214336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828878.754857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.754867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.754873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333213440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828878.754881:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828878.754907:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.754918:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb223b80 x1874663333213568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.754943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333213568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.754949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.754952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.754954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828878.754981:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828878.755010:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb223800 x1874663333213696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333213696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755033:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755035:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755037:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828878.755050:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828878.755066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755075:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb221500 x1874663333213824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333213824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755117:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755120:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828878.755127:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828878.755148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755157:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb221180 x1874663333213952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755166:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333213952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755175:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755177:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333213952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828878.755181:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828878.755195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755201:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb222680 x1874663333214080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755209:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333214080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755215:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828878.755223:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828878.755306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755317:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb220e00 x1874663333214208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755329:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333214208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755358:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755362:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828878.755369:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828878.755392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828878.755403:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff9057fb220a80 x1874663333214336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828878.755411:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333214336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828878.755417:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828878.755420:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828878.755422:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333214336:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828879.138452:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828880.162457:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828881.186417:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828882.210706:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828883.234393:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828883.874326:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828883.874348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874363:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874378:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874388:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828883.874447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333214464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828883.874480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874485:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874489:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874495:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828883.874508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828883.874524:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333214464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828883.874526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333214592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874542:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828883.874549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874553:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874558:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874562:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828883.874568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828883.874591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333214720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874599:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874603:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828883.874605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874609:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874622:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828883.874630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828883.874635:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828883.874643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333214848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874660:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828883.874664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874668:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874672:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874675:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828883.874678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828883.874687:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333214976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828883.874698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874701:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874704:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828883.874707:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158c000 x1874663333214464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828883.874722:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828883.874730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828883.874740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333215104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828883.874746:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333214464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828883.874750:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828883.874761:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874767:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:1.0:1787828883.874767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828883.874774:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828883.874776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828883.874780:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828883.874784:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333214464:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828883.874786:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828883.874797:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333215232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828883.874805:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828883.874808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828883.874809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828883.874811:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10694 00000100:00080000:0.0:1787828883.874813:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828883.874828:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828883.874828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214592:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787828883.874836:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828883.874846:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333215360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828883.874859:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.874873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158f800 x1874663333214592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.874894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333214592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.874901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.874903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.874906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828883.874909:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828883.874940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.874948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158e680 x1874663333214720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.874965:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333214720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.874971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.874974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.874976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828883.874981:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828883.875012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.875018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158df80 x1874663333214848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.875026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333214848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.875031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.875032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.875034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828883.875038:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828883.875048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.875054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158ca80 x1874663333214976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.875061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333214976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.875071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.875073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.875076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333214976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828883.875079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828883.875090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.875096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158f100 x1874663333215104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.875121:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333215104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.875127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.875129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.875131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828883.875134:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828883.875146:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.875152:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158e300 x1874663333215232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.875159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333215232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.875165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.875183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.875186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828883.875211:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828883.875224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828883.875230:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828883] req@ffff9057f158dc00 x1874663333215360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828883.875238:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333215360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828883.875243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828883.875245:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828883.875246:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333215360:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828884.258471:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828885.282383:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828886.306453:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828887.330509:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828888.354538:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828888.994455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828888.994474:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.994487:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994501:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994510:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828888.994567:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333215488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828888.994634:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828888.994640:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.994645:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994649:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994654:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828888.994665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828888.994677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333215616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828888.994688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994696:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828888.994698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333215488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828888.994699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.994712:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994722:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994727:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828888.994731:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828888.994744:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333215744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828888.994753:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994756:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828888.994758:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.994761:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994792:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994797:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828888.994802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828888.994803:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828888.994821:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333215872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828888.994832:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828888.994837:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.994843:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994847:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994854:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828888.994857:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb220a80 x1874663333215488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828888.994860:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828888.994874:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333216000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828888.994891:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.994895:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828888.994896:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828888.994902:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333215488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828888.994908:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.994914:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.994925:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828888.994934:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828888.994949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828888.994950:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333216128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828888.994970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828888.994985:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828888.994987:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333215488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828888.994993:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828888.994998:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.995005:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10699 00000100:00080000:3.0:1787828888.995008:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.995011:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828888.995016:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828888.995026:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333216256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828888.995033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215616:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828888.995049:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828888.995053:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828888.995055:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828888.995062:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10699 00000800:00000100:2.0:1787828888.995072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828888.995073:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828888.995080:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828888.995086:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828888.995091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb220e00 x1874663333215616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828888.995100:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333216384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828888.995128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333215616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995146:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995150:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828888.995157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828888.995175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb222680 x1874663333215744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333215744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995201:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995203:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828888.995210:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828888.995231:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995275:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb221180 x1874663333215872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995284:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333215872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995301:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995304:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333215872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828888.995311:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828888.995334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995370:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb221500 x1874663333216000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995385:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333216000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995413:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995418:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828888.995428:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828888.995457:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995477:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb223800 x1874663333216128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995488:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333216128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995495:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995497:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995501:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828888.995521:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828888.995548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995557:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb223b80 x1874663333216256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995567:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333216256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995592:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995596:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828888.995603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828888.995621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828888.995638:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff9057fb220000 x1874663333216384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.995648:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333216384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.995655:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.995658:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828888.995660:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333216384:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828889.378498:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828890.402508:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828891.426450:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828892.450460:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828893.474358:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828894.114310:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114319:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828894.114322:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114331:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114341:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828894.114347:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828894.114381:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333216512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114406:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114409:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828894.114411:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114414:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114416:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828894.114419:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114426:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828894.114434:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333216640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114442:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114444:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828894.114446:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828894.114453:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828894.114463:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333216768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114470:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828894.114475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114478:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114480:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828894.114485:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828894.114486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333216512:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828894.114489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828894.114500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333216896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828894.114511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114513:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114516:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828894.114518:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828894.114531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333217024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114538:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114558:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828894.114560:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114563:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114567:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828894.114570:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828894.114574:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828894.114594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333217152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828894.114606:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828894.114613:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158d880 x1874663333216512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828894.114614:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114626:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114633:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828894.114637:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333216512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828894.114642:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114653:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828894.114655:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114661:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.114665:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333216512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787828894.114670:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333217280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828894.114676:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216640:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828894.114686:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828894.114689:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828894.114691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828894.114696:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10704 00000100:00080000:0.0:1787828894.114700:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828894.114705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828894.114706:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828894.114714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828894.114721:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158ce00 x1874663333216640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828894.114727:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333217408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828894.114744:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333216640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.114754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.114760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828894.114765:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828894.114779:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.114786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158c700 x1874663333216768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.114799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333216768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.114804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.114819:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828894.114842:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828894.114865:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.114878:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158ea00 x1874663333216896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.114893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333216896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.114902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.114910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333216896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828894.114915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828894.114931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.114939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158d180 x1874663333217024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.114947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333217024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.114958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.114963:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828894.114967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828894.114977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.114984:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158d500 x1874663333217152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.114991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333217152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.114997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.114999:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.115001:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828894.115005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828894.115015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.115021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158ed80 x1874663333217280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.115029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333217280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.115034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.115036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.115039:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828894.115043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828894.115054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828894.115060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9057f158fb80 x1874663333217408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828894.115068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333217408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828894.115073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828894.115075:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828894.115077:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333217408:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828894.498401:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828895.522368:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828896.546447:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828897.570390:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828898.594355:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828899.234326:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828899.234341:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234349:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234366:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828899.234400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333217536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234420:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234422:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828899.234424:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234427:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234429:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234431:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234438:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828899.234447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333217664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828899.234468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234475:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828899.234487:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333217792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828899.234489:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333217536:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828899.234499:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828899.234503:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234519:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234521:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234524:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828899.234537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333217920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234544:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828899.234558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234564:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234566:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828899.234577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333218048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828899.234591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828899.234593:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828899.234594:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234599:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234602:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828899.234612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333218176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828899.234623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234626:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234628:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828899.234630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234633:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828899.234639:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333218304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828899.234646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828899.234648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828899.234649:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828899.234653:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10709 00000100:00080000:0.0:1787828899.234655:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828899.234655:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158fb80 x1874663333217536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828899.234660:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828899.234666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828899.234676:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333218432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828899.234686:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333217536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.234698:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.234704:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.234709:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333217536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828899.234719:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828899.234758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.234773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158ed80 x1874663333217664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.234786:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333217664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.234795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.234799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.234802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828899.234825:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828899.234845:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.234855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158d500 x1874663333217792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.234866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333217792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.234874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.234877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.234880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828899.234885:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828899.234901:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.234910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158d180 x1874663333217920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.234930:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333217920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.234947:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.234950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.234954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333217920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828899.234960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828899.234977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.234986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158ea00 x1874663333218048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.235013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333218048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.235022:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.235025:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.235028:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828899.235034:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828899.235051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.235071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158c700 x1874663333218176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.235084:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333218176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.235099:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.235102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.235105:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828899.235111:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828899.235126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.235147:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158ce00 x1874663333218304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.235158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333218304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.235166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.235170:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.235173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828899.235179:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828899.235196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828899.235214:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828899/real 1787828899] req@ffff9057f158d880 x1874663333218432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828899.235259:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333218432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828899.235278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828899.235281:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828899.235285:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333218432:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828899.618411:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828900.642375:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828901.666369:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828902.690371:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828903.714488:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828904.354309:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354329:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828904.354332:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354340:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354349:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354356:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828904.354391:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333218560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828904.354411:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354414:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828904.354415:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354418:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354421:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354423:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354431:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828904.354439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333218688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828904.354448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354450:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828904.354452:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828904.354456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333218560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828904.354458:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354464:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354467:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828904.354499:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333218816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828904.354516:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354519:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828904.354520:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354526:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354529:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354533:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828904.354542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333218944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828904.354542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828904.354556:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354559:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828904.354560:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354563:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354567:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828904.354578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333219072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828904.354584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828904.354589:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158dc00 x1874663333218560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828904.354593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354602:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354606:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828904.354621:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333218560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828904.354623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333219200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828904.354646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828904.354647:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354653:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828904.354653:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828904.354658:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828904.354660:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333218560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828904.354664:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354669:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354674:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828904.354680:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333219328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828904.354686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218688:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828904.354691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828904.354694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828904.354696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828904.354699:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10715 00000100:00080000:0.0:1787828904.354703:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828904.354705:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828904.354710:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828904.354711:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828904.354724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333219456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828904.354725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158e300 x1874663333218688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333218688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354770:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354774:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828904.354780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828904.354804:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.354816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158f100 x1874663333218816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333218816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828904.354841:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828904.354855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.354862:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158ca80 x1874663333218944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333218944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333218944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828904.354881:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828904.354891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.354896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158df80 x1874663333219072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333219072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354915:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828904.354920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828904.354930:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.354935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158e680 x1874663333219200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354942:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333219200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354947:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828904.354963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828904.354973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.354979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158f800 x1874663333219328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.354986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333219328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.354995:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.354997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.354999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828904.355002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828904.355012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828904.355028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828904/real 1787828904] req@ffff9057f158c000 x1874663333219456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828904.355035:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333219456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828904.355040:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828904.355041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828904.355043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333219456:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828904.738374:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828905.762448:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828906.786386:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828907.810413:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828908.834369:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828909.474339:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474347:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828909.474351:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474359:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474368:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474374:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828909.474405:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333219584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828909.474456:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474460:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474462:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474465:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828909.474483:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333219712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474492:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828909.474496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333219584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787828909.474496:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828909.474499:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474507:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474511:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828909.474523:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333219840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828909.474533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474536:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474539:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474541:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828909.474553:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333219968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828909.474562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474564:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474566:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474568:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828909.474580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333220096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474586:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828909.474590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474595:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00100000:1.0:1787828909.474596:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219712:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787828909.474598:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474604:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474608:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828909.474618:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333220224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828909.474629:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219840:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787828909.474632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828909.474636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828909.474637:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474641:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474645:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828909.474650:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219968:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787828909.474650:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474658:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828909.474670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333220352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828909.474677:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828909.474679:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220096:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787828909.474682:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828909.474685:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828909.474690:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10720 00000100:00080000:2.0:1787828909.474693:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828909.474696:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828909.474700:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828909.474704:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828909.474710:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333220480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828909.474727:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828909.474744:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220480:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828909.858476:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828910.882467:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828911.906405:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828912.546479:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787828912.546524:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828912.546542:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546581:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546590:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546598:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546605:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546614:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546621:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787828912.546634:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787828912.546698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de0e00 x1874663333220480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546739:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333220480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546752:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546762:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828912.546770:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de1f80 x1874663333220224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546785:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333220224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546801:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.546808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de3100 x1874663333220352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333220352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546835:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333220352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.546845:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de2a00 x1874663333219712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546853:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333219712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546861:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546864:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.546874:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de1180 x1874663333219840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333219840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828912.546916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de1880 x1874663333219968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333219968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546930:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333219968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828912.546938:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de3800 x1874663333220096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333220096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546952:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828912.546956:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828909/real 1787828912] req@ffff9057d0de0700 x1874663333219584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828912.546963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333219584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828912.546967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828912.546969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828912.546971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333219584:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1787828912.930400:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828913.954426:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828914.594424:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828914.594445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594458:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.594472:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594480:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828914.594520:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333220608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.594547:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828914.594551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594555:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.594558:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594560:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828914.594570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828914.594584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333220736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.594598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828914.594604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594613:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00100000:1.0:1787828914.594615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333220608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828914.594620:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828914.594640:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828914.594665:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333220864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.594677:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594681:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828914.594683:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594688:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.594693:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594698:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828914.594705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828914.594708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828914.594730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333220992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.594742:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828914.594750:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594754:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.594758:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594762:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828914.594769:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828914.594784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158c000 x1874663333220608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828914.594788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333221120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828914.594828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333220608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828914.594830:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594843:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828914.594845:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828914.594861:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:1.0:1787828914.594864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828914.594867:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828914.594877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787828914.594879:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828914.594889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333220608:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787828914.594891:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828914.594916:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333221248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.594932:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.594936:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828914.594940:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828914.594943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787828914.594951:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.594958:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.594964:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828914.594979:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828914.594993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787828914.595001:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333221376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828914.595027:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828914.595031:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828914.595034:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828914.595041:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158f800 x1874663333220736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828914.595048:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10725 00000100:00080000:0.0:1787828914.595063:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828914.595073:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828914.595080:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828914.595082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333220736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828914.595106:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333221504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828914.595114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.595144:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828914.595183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595205:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158e680 x1874663333220864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595259:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333220864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595279:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595283:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.595291:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828914.595318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595331:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158df80 x1874663333220992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595342:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333220992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595354:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595357:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333220992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828914.595365:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828914.595386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595412:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158ca80 x1874663333221120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595427:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333221120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595439:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595443:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828914.595450:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828914.595475:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595487:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158f100 x1874663333221248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595500:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333221248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595512:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595515:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.595542:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828914.595566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595578:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158e300 x1874663333221376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595598:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333221376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595608:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595611:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595615:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.595622:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828914.595643:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828914.595656:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff9057f158dc00 x1874663333221504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.595668:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333221504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.595675:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.595679:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828914.595682:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333221504:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828914.978361:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828916.002399:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828917.026404:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828918.050514:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828919.074517:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828919.714321:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714329:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828919.714333:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714340:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714349:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714355:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828919.714398:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333221632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714435:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828919.714437:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714446:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714449:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714457:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828919.714466:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333221760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.714477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714480:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714485:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828919.714491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333221632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828919.714500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333221888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714508:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828919.714512:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714514:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714516:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714519:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828919.714531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333222016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714537:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.714540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714542:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714544:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714549:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787828919.714550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714553:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828919.714561:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333222144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.714570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828919.714572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714575:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714578:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.714580:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828919.714588:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb222a00 x1874663333221632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828919.714596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333222272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.714605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828919.714609:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333221632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828919.714610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.714614:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00080000:3.0:1787828919.714626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828919.714627:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828919.714629:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828919.714634:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787828919.714635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828919.714639:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333221632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828919.714645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333222400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828919.714648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221760:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828919.714666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828919.714671:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828919.714672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.714677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.714684:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10730 00000100:00100000:1.0:1787828919.714685:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb220700 x1874663333221760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828919.714691:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828919.714699:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333221760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828919.714702:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828919.714713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828919.714716:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714723:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828919.714732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333222528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828919.714746:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828919.714762:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714772:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb221880 x1874663333221888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714779:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333221888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714789:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333221888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828919.714792:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828919.714812:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb223100 x1874663333222016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333222016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714831:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828919.714837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828919.714847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb222d80 x1874663333222144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333222144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828919.714871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828919.714881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714886:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb221c00 x1874663333222272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333222272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714898:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828919.714905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828919.714915:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714921:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb222300 x1874663333222400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714928:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333222400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714933:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714934:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714936:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828919.714947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828919.714957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828919.714963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff9057fb220380 x1874663333222528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.714969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333222528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.714974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.714976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828919.714978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333222528:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828920.098409:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828921.122558:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828922.146465:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828923.170504:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828924.194422:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828924.834287:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834299:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828924.834305:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834318:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834347:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834357:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828924.834409:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333222656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828924.834438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828924.834443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834463:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828924.834474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333222784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828924.834482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834485:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828924.834486:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834489:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834491:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834493:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828924.834500:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333222656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828924.834507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333222912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828924.834521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828924.834527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834532:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834540:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828924.834560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333223040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828924.834571:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828924.834578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834581:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834586:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828924.834587:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828924.834594:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834598:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828924.834608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333223168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828924.834616:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834619:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828924.834620:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828924.834655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333223296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828924.834662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb220380 x1874663333222656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828924.834664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828924.834677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834687:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834694:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834699:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828924.834702:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333222656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828924.834708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828924.834715:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.834719:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.834725:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333222656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828924.834732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333223424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828924.834736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828924.834749:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828924.834753:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828924.834755:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828924.834758:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10735 00000100:00080000:3.0:1787828924.834764:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828924.834768:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828924.834773:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828924.834777:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828924.834787:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333223552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828924.834797:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb222300 x1874663333222784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.834815:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333222784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.834822:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.834832:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.834835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.834860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828924.834877:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.834884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb221c00 x1874663333222912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.834893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333222912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.834903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.834906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.834908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333222912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.834911:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828924.834923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.834929:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb222d80 x1874663333223040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.834936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333223040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.834942:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.834943:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.834945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828924.834949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828924.834960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.834965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb223100 x1874663333223168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.834973:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333223168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.834978:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.834980:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.834982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828924.834986:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828924.834998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.835003:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb221880 x1874663333223296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.835010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333223296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.835016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.835017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.835019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.835022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828924.835032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.835038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb220700 x1874663333223424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.835044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333223424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.835049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.835052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.835053:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.835056:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828924.835066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828924.835073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff9057fb222a00 x1874663333223552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.835080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333223552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.835086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.835087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828924.835089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333223552:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828925.218599:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828926.242473:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828927.266450:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828928.290413:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828929.314400:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828929.954312:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954319:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828929.954323:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954330:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954339:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954344:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828929.954378:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333223680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.954401:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954403:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828929.954405:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954407:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954410:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954412:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954419:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828929.954426:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333223808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.954433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954435:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828929.954437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954439:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954442:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954444:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954448:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828929.954454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333223936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.954462:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828929.954463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333223680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787828929.954466:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828929.954471:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954475:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954477:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954479:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828929.954492:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333224064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.954498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954501:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828929.954502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954505:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954507:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954509:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954512:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828929.954519:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333224192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828929.954527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828929.954528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828929.954534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954538:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954542:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954547:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828929.954566:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de1500 x1874663333223680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828929.954567:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333224320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.954591:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828929.954594:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333223680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828929.954598:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828929.954602:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828929.954611:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828929.954611:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00080000:2.0:1787828929.954616:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828929.954617:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787828929.954625:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954626:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333223680:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787828929.954635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828929.954637:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828929.954651:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333224448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828929.954659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828929.954661:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.954665:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828929.954666:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.954671:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10740 00000100:00100000:1.0:1787828929.954672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de1c00 x1874663333223808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828929.954674:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.954681:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828929.954688:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828929.954701:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333223808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828929.954701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333224576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828929.954735:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954739:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954744:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.954762:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828929.954779:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de0000 x1874663333223936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333223936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954800:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954804:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333223936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.954807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828929.954818:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954824:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de0a80 x1874663333224064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954831:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333224064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954835:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954837:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954839:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828929.954843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828929.954853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954858:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de0700 x1874663333224192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333224192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828929.954876:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828929.954886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de3800 x1874663333224320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333224320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.954909:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828929.954919:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954925:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de1880 x1874663333224448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333224448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.954944:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828929.954953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828929.954958:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff9057d0de1180 x1874663333224576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.954964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333224576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.954969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.954971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828929.954973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333224576:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828930.338433:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828931.362406:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828932.386419:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828933.410429:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828934.434343:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828935.074352:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074360:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828935.074364:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074381:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074388:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828935.074421:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333224704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.074450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828935.074456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074464:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074469:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828935.074495:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333224704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828935.074496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333224832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.074510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074512:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.074514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074520:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074522:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828935.074537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333224960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.074545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828935.074549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074551:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074556:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074559:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828935.074566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828935.074573:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333225088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.074581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.074585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074587:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074589:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074592:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074595:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828935.074605:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333225216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828935.074610:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb220000 x1874663333224704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828935.074612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074619:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828935.074623:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074630:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074634:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074640:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828935.074655:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333224704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828935.074656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333225344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828935.074685:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074690:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828935.074697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828935.074698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333224704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828935.074703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.074706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.074709:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074712:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074715:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074718:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828935.074723:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828935.074728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333225472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.074738:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.074740:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.074742:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828935.074746:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828935.074747:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10745 00000100:00080000:3.0:1787828935.074754:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.074757:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828935.074761:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828935.074765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb223b80 x1874663333224832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828935.074773:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333225600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828935.074780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333224832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074797:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828935.074802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828935.074814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.074820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb223800 x1874663333224960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.074828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333224960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333224960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828935.074844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828935.074855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.074860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb221500 x1874663333225088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.074867:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333225088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074874:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074876:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828935.074879:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828935.074888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.074893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb221180 x1874663333225216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.074900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333225216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828935.074912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828935.074921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.074927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb222680 x1874663333225344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.074933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333225344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828935.074953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828935.074963:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.074969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb220e00 x1874663333225472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.074976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333225472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.074980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.074982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.074984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828935.074987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828935.074996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828935.075002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9057fb220a80 x1874663333225600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828935.075009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333225600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828935.075014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828935.075016:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828935.075018:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333225600:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828935.458414:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828936.482409:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828937.506476:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828938.530396:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828939.554452:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828940.194315:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194322:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828940.194326:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194334:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194343:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194349:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828940.194384:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333225728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194404:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194407:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828940.194408:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194412:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194414:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194416:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194423:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828940.194432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333225856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828940.194442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194444:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194446:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194448:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194452:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828940.194458:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333225984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828940.194469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194472:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194473:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828940.194478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333225728:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828940.194479:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194487:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828940.194494:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333226112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194501:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828940.194504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194506:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194511:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194514:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828940.194521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333226240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194531:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828940.194532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194534:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194538:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828940.194542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787828940.194544:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828940.194557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333226368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194566:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828940.194567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194571:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194574:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194577:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828940.194580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828940.194581:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158d880 x1874663333225728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.194589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333226496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828940.194600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828940.194604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828940.194604:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333225728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.194605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828940.194613:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10750 00000100:00080000:0.0:1787828940.194619:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.194624:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828940.194626:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828940.194627:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828940.194635:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787828940.194639:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333226624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828940.194641:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333225728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787828940.194656:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828940.194674:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158ce00 x1874663333225856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194690:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333225856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194700:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194702:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194705:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828940.194708:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828940.194718:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194724:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158c700 x1874663333225984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333225984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194736:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194738:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194739:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333225984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828940.194757:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828940.194768:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194774:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158ea00 x1874663333226112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333226112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194789:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828940.194793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828940.194803:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194808:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158d180 x1874663333226240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194815:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333226240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194820:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194822:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828940.194827:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828940.194836:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158d500 x1874663333226368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333226368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194854:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828940.194859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828940.194869:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158ed80 x1874663333226496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333226496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194888:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828940.194893:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828940.194903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828940.194908:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9057f158fb80 x1874663333226624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828940.194914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333226624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.194919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.194921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828940.194923:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333226624:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828940.578402:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828941.602392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828942.626397:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828943.650406:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828944.674481:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828945.314354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314367:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828945.314373:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828945.314386:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314400:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314409:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828945.314462:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333226752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314502:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828945.314505:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828945.314509:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314512:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314516:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314528:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828945.314549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333226880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314560:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828945.314568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828945.314573:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314576:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314580:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314584:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828945.314594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333227008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314608:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828945.314610:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828945.314642:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00100000:1.0:1787828945.314645:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333226752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787828945.314646:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314654:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314668:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828945.314682:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333227136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828945.314717:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828945.314722:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314727:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314731:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314739:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828945.314753:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333227264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314783:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314787:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828945.314789:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828945.314796:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828945.314797:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314805:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314809:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314814:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828945.314831:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333227392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828945.314845:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.314851:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828945.314854:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828945.314863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de2a00 x1874663333226752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828945.314864:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.314875:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828945.314888:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.314903:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828945.314910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333226752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.314926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.314930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.314941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333226752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828945.314943:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333227520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828945.314953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226880:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787828945.314993:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828945.315001:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828945.315006:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828945.315012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787828945.315022:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10756 00000100:00080000:2.0:1787828945.315032:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828945.315034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de3100 x1874663333226880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828945.315062:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787828945.315075:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828945.315082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333226880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315095:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828945.315098:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333227648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828945.315104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315115:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333226880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828945.315129:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828945.315153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315163:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de1f80 x1874663333227008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315175:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333227008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315259:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315263:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828945.315293:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828945.315336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315351:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de0e00 x1874663333227136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315366:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333227136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315381:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315384:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828945.315410:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828945.315430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315440:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de1180 x1874663333227264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315453:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333227264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315462:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315466:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315470:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828945.315475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828945.315493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315502:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de1880 x1874663333227392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315514:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333227392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315526:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315529:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828945.315535:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828945.315554:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315571:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de3800 x1874663333227520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315591:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333227520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315625:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315630:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828945.315639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828945.315669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828945.315682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828945/real 1787828945] req@ffff9057d0de0700 x1874663333227648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828945.315713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333227648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828945.315723:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828945.315726:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828945.315730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333227648:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828945.698438:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828946.722449:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828947.746363:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828948.770368:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828949.794358:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828950.434365:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434381:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828950.434386:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434400:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434415:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434424:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828950.434473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333227776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434499:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828950.434507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434511:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434517:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828950.434537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333227904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828950.434550:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333227776:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828950.434550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828950.434554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434562:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434564:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434567:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828950.434590:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333228032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828950.434602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434604:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434607:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434610:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434614:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828950.434622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333228160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434631:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828950.434633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828950.434637:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828950.434639:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434643:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434646:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828950.434657:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333228288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828950.434667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434671:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434676:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434679:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828950.434690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333228416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828950.434696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828950.434704:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb220a80 x1874663333227776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828950.434704:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828950.434709:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434715:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434721:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828950.434731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333227776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828950.434736:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333228544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828950.434756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828950.434757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828950.434764:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828950.434765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828950.434769:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828950.434773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333227776:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828950.434777:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10761 00000100:00080000:3.0:1787828950.434781:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828950.434784:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828950.434788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828950.434794:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333228672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828950.434801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828950.434824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.434837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb220e00 x1874663333227904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.434846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333227904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.434857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.434859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.434862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333227904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828950.434865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828950.434877:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.434883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb222680 x1874663333228032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.434894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333228032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.434903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.434907:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.434910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828950.434916:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828950.434939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.434950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb221180 x1874663333228160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.434958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333228160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.434980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.434982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.434986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828950.434991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828950.435007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.435014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb221500 x1874663333228288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.435022:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333228288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.435028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.435030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.435032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828950.435036:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828950.435049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.435056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb223800 x1874663333228416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.435064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333228416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.435070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.435072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.435074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828950.435077:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828950.435089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.435095:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb223b80 x1874663333228544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.435113:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333228544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.435119:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.435122:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.435124:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828950.435142:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828950.435189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828950.435198:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828950/real 1787828950] req@ffff9057fb220000 x1874663333228672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828950.435206:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333228672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828950.435212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828950.435214:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828950.435216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333228672:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828950.818390:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828951.842401:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828952.866372:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828953.890462:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828954.914407:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828955.554325:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554336:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828955.554339:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554362:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554370:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828955.554412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333228800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554442:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828955.554443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554449:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828955.554470:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333228928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554478:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554481:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828955.554483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554489:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554496:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828955.554507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333229056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554518:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828955.554519:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333228800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828955.554523:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828955.554527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554537:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554540:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828955.554556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333229184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554563:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828955.554567:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554573:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554576:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828955.554587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333229312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828955.554602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554608:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10766 00000800:00000100:1.0:1787828955.554613:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828955.554616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554621:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554624:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828955.554647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333229440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554658:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828955.554662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb222a00 x1874663333228800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828955.554664:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828955.554669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828955.554681:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:3.0:1787828955.554689:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828955.554701:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554711:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828955.554728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333228800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828955.554730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333229568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828955.554763:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828955.554767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828955.554770:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828955.554771:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828955.554775:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10766 00000100:00080000:1.0:1787828955.554780:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828955.554786:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828955.554793:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333228800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828955.554798:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828955.554811:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828955.554813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828955.554837:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333229696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828955.554861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.554875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb220700 x1874663333228928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.554883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333228928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.554889:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.554892:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.554894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333228928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828955.554920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828955.554942:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.554953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb221880 x1874663333229056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.554962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333229056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.554971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.554974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.554976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828955.554980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828955.554995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.555001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb223100 x1874663333229184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.555008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333229184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.555014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.555016:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.555017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828955.555021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828955.555032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.555037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb222d80 x1874663333229312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.555044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333229312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.555049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.555051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.555054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828955.555058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828955.555068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.555074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb221c00 x1874663333229440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.555081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333229440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.555090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.555092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.555096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828955.555101:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828955.555127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.555136:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb222300 x1874663333229568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.555144:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333229568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.555150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.555152:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.555154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828955.555160:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828955.555177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828955.555184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828955/real 1787828955] req@ffff9057fb220380 x1874663333229696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828955.555191:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333229696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828955.555197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828955.555198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828955.555200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333229696:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828955.938504:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828956.962406:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828957.986483:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828959.010402:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828960.034522:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828960.674305:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674316:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828960.674320:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674341:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674352:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674359:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828960.674398:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333229824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828960.674425:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674429:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828960.674431:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674435:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674438:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674441:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828960.674450:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828960.674461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333229952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828960.674479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674483:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828960.674486:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674489:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674492:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674495:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828960.674499:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828960.674508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333230080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828960.674552:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333229824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828960.674553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828960.674562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674565:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828960.674574:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828960.674583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333230208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828960.674589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674603:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828960.674605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674607:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674609:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674611:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828960.674615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828960.674624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333230336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828960.674627:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828960.674639:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674643:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828960.674645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674650:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674655:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674674:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb220380 x1874663333229824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828960.674679:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828960.674698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333230464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828960.674699:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333229824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828960.674710:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828960.674713:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828960.674714:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828960.674718:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674723:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674729:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787828960.674734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828960.674734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828960.674743:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787828960.674743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333230592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828960.674760:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828960.674761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333229824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828960.674765:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828960.674768:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828960.674773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229952:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828960.674774:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10771 00000100:00080000:3.0:1787828960.674778:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828960.674782:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828960.674786:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828960.674795:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333230720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828960.674804:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb222300 x1874663333229952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.674826:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333229952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.674844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.674846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.674849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333229952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.674865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828960.674875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb221c00 x1874663333230080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.674889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333230080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.674894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.674895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.674897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.674900:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828960.674911:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb222d80 x1874663333230208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.674934:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333230208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.674939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.674940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.674943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828960.674946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828960.674956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb223100 x1874663333230336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.674968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333230336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.674973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.674974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.674976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787828960.674979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828960.674988:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.674994:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb221880 x1874663333230464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.675000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333230464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.675005:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.675017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.675020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.675023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787828960.675032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.675038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb220700 x1874663333230592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.675045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333230592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.675050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.675052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.675053:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.675057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787828960.675066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828960.675071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff9057fb222a00 x1874663333230720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.675091:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333230720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.675097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.675099:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787828960.675101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333230720:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828961.058457:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828962.082401:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828963.106343:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828964.130457:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828965.154360:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828965.794324:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794335:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828965.794339:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794361:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794369:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828965.794412:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333230848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794439:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828965.794443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794447:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794451:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828965.794470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333230976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794481:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828965.794482:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794484:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794486:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794488:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794491:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828965.794501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333231104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828965.794503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333230848:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828965.794516:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828965.794521:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794524:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794525:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794527:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828965.794539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333231232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794545:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828965.794548:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794550:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794552:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794554:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828965.794564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333231360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794569:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828965.794573:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794577:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794580:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794582:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794585:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828965.794591:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787828965.794595:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333231488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794603:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794605:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828965.794607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794610:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794612:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828965.794615:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794618:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828965.794625:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333231616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.794632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828965.794634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828965.794635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828965.794638:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10776 00000100:00080000:1.0:1787828965.794641:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828965.794645:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78aaa00 x1874663333230848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828965.794648:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828965.794656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828965.794669:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333230848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828965.794670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333231744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828965.794699:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.794704:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.794708:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333230848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828965.794715:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828965.794732:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.794742:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78ab480 x1874663333230976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.794750:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333230976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.794756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.794759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.794761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333230976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828965.794797:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828965.794818:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.794831:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78abb80 x1874663333231104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.794845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333231104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.794854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.794858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.794862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828965.794869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828965.794891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.794904:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78a8000 x1874663333231232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.794915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333231232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.794924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.794928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.794931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828965.794938:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828965.794959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.794972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78aad80 x1874663333231360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.794985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333231360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.794994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.794997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.794999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333231360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828965.795004:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828965.795021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.795028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78a9880 x1874663333231488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.795036:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333231488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.795057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.795059:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.795062:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828965.795067:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828965.795082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.795089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78ab800 x1874663333231616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.795097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333231616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.795102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.795104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.795106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828965.795110:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828965.795120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828965.795126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff9057c78a8380 x1874663333231744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.795133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333231744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.795139:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.795140:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828965.795142:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333231744:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828966.178455:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828967.202413:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828968.226609:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828969.250384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828970.274465:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828970.914381:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914395:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828970.914400:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914415:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914431:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914443:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828970.914509:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333231872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828970.914552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914557:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914562:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914567:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.914578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828970.914595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333232000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914619:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828970.914622:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828970.914630:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333231872:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828970.914631:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914639:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914649:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.914654:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828970.914668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333232128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914678:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914682:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828970.914684:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914688:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914692:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914695:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.914701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828970.914712:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333232256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914722:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914725:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828970.914728:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914731:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914736:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914739:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.914744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828970.914752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787828970.914760:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333232384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914772:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914776:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828970.914778:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914784:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914798:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914802:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.914816:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828970.914838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb220000 x1874663333231872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828970.914838:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333232512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828970.914878:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914886:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828970.914888:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828970.914894:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914899:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914903:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828970.914911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333231872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828970.914911:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828970.914929:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333232640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828970.914944:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.914960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828970.914961:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828970.914969:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828970.914975:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828970.914976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333231872:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828970.914985:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10781 00000100:00080000:3.0:1787828970.914992:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828970.914996:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828970.915001:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828970.915036:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333232768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828970.915071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828970.915120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915145:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb223b80 x1874663333232000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915168:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333232000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915184:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915196:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828970.915207:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828970.915309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915328:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb223800 x1874663333232128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333232128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915379:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915384:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828970.915394:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828970.915429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915450:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb221500 x1874663333232256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915470:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333232256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915489:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915495:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828970.915505:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828970.915540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915559:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb221180 x1874663333232384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915579:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333232384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915604:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915615:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333232384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828970.915625:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828970.915657:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915677:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb222680 x1874663333232512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915697:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333232512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915710:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915716:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915721:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828970.915732:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828970.915765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915802:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb220e00 x1874663333232640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333232640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915845:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828970.915889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828970.915928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828970.915949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff9057fb220a80 x1874663333232768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.915972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333232768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828970.915987:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828970.915992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828970.915999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333232768:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828971.298488:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828972.322406:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828973.346376:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828974.370380:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828975.394467:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828976.034268:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828976.034280:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828976.034284:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034294:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034305:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034313:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828976.034353:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333232896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034383:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828976.034386:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828976.034388:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034393:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034406:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034410:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034429:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828976.034445:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333233024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034456:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828976.034459:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333232896:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828976.034460:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828976.034462:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034469:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828976.034501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333233152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034512:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828976.034516:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828976.034518:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034527:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034530:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828976.034545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333233280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828976.034555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828976.034557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828976.034561:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233024:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828976.034564:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034575:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828976.034586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333233408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828976.034593:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233152:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828976.034596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828976.034600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828976.034602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034606:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034612:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828976.034616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233280:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787828976.034618:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828976.034631:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333233536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034640:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828976.034642:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233408:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787828976.034644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828976.034647:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034655:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034657:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828976.034669:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828976.034674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333233664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828976.034684:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828976.034702:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233664:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828976.034706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828976.034709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828976.034713:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10786 00000100:00080000:3.0:1787828976.034715:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828976.034718:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828976.034722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828976.034733:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333233792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828976.034753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233792:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828976.418432:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828977.442380:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828978.466382:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828979.106491:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1787828979.106528:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828979.106543:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106584:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106593:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106601:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106609:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106617:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106628:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1787828979.106638:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787828979.106659:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb223b80 x1874663333233664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106686:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333233664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106698:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106703:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106707:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828979.106715:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb223800 x1874663333233536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106727:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333233536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106735:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106738:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106742:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828979.106749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb221500 x1874663333233408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333233408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106770:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828979.106780:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb220e00 x1874663333233024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333233024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106799:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828979.106812:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb222680 x1874663333233152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106823:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333233152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828979.106842:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb221180 x1874663333233280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333233280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106859:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106862:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106866:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333233280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828979.106872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb220a80 x1874663333232896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333232896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333232896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828979.106930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828979] req@ffff9057fb220000 x1874663333233792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828979.106943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333233792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828979.106969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828979.106972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828979.106975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333233792:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787828979.490391:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828980.514356:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828981.154414:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828981.154432:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.154443:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154455:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154463:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787828981.154504:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333233920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.154550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154554:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.154556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.154561:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154565:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154570:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154580:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828981.154601:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333234048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828981.154601:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333233920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828981.154626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154630:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.154633:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.154637:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154642:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154652:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828981.154664:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333234176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.154675:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154679:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.154683:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828981.154684:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828981.154691:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154696:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154702:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154707:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828981.154720:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333234304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.154731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828981.154736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78a9f80 x1874663333233920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828981.154740:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.154746:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154752:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154759:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154766:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828981.154771:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333233920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828981.154782:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333234432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828981.154804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828981.154807:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828981.154813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828981.154814:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.154820:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828981.154824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333233920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787828981.154829:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154835:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.154839:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154845:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828981.154856:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787828981.154860:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333234560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.154874:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.154878:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.154880:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.154885:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154890:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828981.154893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787828981.154898:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787828981.154905:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828981.154914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78a8380 x1874663333234048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828981.154920:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333234688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828981.154958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333234048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828981.154972:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828981.154975:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828981.154978:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.154983:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828981.154985:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787828981.154991:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10791 00000100:00080000:1.0:1787828981.154997:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828981.154999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234048:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787828981.155005:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234176:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787828981.155014:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828981.155034:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333234816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828981.155046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155062:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78ab800 x1874663333234176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155076:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333234176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155084:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828981.155097:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828981.155116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78a9880 x1874663333234304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155136:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333234304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155147:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155151:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828981.155157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828981.155176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155187:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78aad80 x1874663333234432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155198:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333234432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333234432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828981.155290:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828981.155318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155332:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78a8000 x1874663333234560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155345:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333234560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155357:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155361:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828981.155367:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828981.155387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155404:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78abb80 x1874663333234688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155416:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333234688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155441:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155445:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828981.155451:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828981.155471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828981.155483:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9057c78ab480 x1874663333234816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828981.155494:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333234816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828981.155507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828981.155510:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828981.155513:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333234816:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828981.538392:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828982.562470:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828983.586389:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828984.610373:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828985.634396:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828986.274327:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274335:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828986.274338:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274346:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274354:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274360:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828986.274389:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333234944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828986.274409:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274413:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828986.274415:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274419:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274423:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274427:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274437:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828986.274450:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333235072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828986.274452:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333234944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828986.274460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828986.274465:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274468:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274471:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274475:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828986.274489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333235200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828986.274497:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828986.274502:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274505:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10797 00000800:00000100:2.0:1787828986.274508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828986.274508:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828986.274524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333235328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828986.274532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828986.274537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274541:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274544:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828986.274546:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb222a00 x1874663333234944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828986.274547:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274552:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828986.274562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333235456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828986.274564:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333234944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828986.274572:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828986.274575:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828986.274575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828986.274577:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828986.274579:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828986.274581:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00100000:2.0:1787828986.274583:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333234944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828986.274584:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274588:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828986.274600:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333235584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828986.274605:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235072:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828986.274610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828986.274623:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828986.274625:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828986.274627:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828986.274628:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274632:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274635:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274639:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb220700 x1874663333235072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828986.274641:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828986.274652:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333235072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274661:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828986.274663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333235712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828986.274664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274669:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235072:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828986.274673:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828986.274675:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235200:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787828986.274677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828986.274679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828986.274682:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10797 00000100:00080000:3.0:1787828986.274685:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828986.274689:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828986.274693:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787828986.274694:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828986.274703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333235840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828986.274704:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb221880 x1874663333235200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274716:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333235200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274725:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828986.274738:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828986.274766:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274775:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb223100 x1874663333235328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333235328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274788:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274790:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828986.274796:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828986.274807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb222d80 x1874663333235456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333235456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274825:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274827:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333235456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828986.274840:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828986.274851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274857:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb221c00 x1874663333235584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333235584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828986.274876:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828986.274886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb222300 x1874663333235712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274898:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333235712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828986.274930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828986.274951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828986.274958:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828986/real 1787828986] req@ffff9057fb220380 x1874663333235840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828986.274966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333235840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828986.274971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828986.274973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828986.274975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333235840:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828986.658430:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828987.682376:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828988.706363:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828989.730433:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828990.754406:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828991.394366:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394376:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828991.394381:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394393:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394404:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394411:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787828991.394454:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333235968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394482:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828991.394484:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394487:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394490:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394494:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828991.394513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333236096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394520:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394522:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828991.394524:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394525:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394528:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394530:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394532:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828991.394538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333236224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828991.394556:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394559:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394561:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828991.394576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333236352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828991.394592:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394597:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394610:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828991.394612:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333235968:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787828991.394623:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828991.394647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333236480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394660:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828991.394661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394666:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394671:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394674:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394677:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828991.394684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333236608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828991.394695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394698:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394702:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394708:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787828991.394712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394715:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828991.394727:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333236736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828991.394736:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828991.394739:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828991.394740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828991.394743:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10802 00000100:00080000:0.0:1787828991.394745:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828991.394748:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787828991.394751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828991.394763:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333236864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828991.394775:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158fb80 x1874663333235968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.394804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333235968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.394817:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.394823:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.394828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333235968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787828991.394836:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828991.394882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.394898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158ed80 x1874663333236096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.394913:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333236096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.394922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.394925:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.394929:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828991.394936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828991.394958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.394970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158d500 x1874663333236224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.394983:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333236224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.394991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.394995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.394999:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828991.395005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828991.395025:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.395034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158d180 x1874663333236352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.395042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333236352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.395047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.395049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.395052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828991.395075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828991.395086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.395093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158ea00 x1874663333236480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.395100:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333236480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.395105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.395107:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.395109:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333236480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828991.395113:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828991.395123:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.395129:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158c700 x1874663333236608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.395151:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333236608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.395156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.395158:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.395160:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828991.395164:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828991.395176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.395182:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158ce00 x1874663333236736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.395189:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333236736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.395194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.395196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.395198:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828991.395204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828991.395274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828991.395288:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828991/real 1787828991] req@ffff9057f158d880 x1874663333236864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828991.395298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333236864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828991.395308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828991.395310:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828991.395312:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333236864:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787828991.778401:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828992.802421:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828993.826373:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828994.850402:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828995.874368:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828996.514351:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514360:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828996.514364:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514374:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00080000:3.0:1787828996.514384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787828996.514437:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333236992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828996.514463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828996.514469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514474:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00080000:3.0:1787828996.514482:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828996.514495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828996.514509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333236992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787828996.514510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333237120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828996.514526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828996.514531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514535:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00080000:3.0:1787828996.514539:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514543:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828996.514549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828996.514560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333237248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828996.514569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828996.514576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828996.514582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514586:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00080000:3.0:1787828996.514592:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514596:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828996.514601:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828996.514613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333237376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828996.514618:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb220380 x1874663333236992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828996.514644:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333236992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828996.514649:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828996.514658:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828996.514659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828996.514664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787828996.514668:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00100000:2.0:1787828996.514671:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333236992:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787828996.514674:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514681:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787828996.514685:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828996.514693:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828996.514699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333237504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828996.514711:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828996.514717:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828996.514718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828996.514722:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514728:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00100000:2.0:1787828996.514732:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb222300 x1874663333237120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828996.514733:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514743:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828996.514748:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333237120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828996.514750:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828996.514760:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.514764:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.514767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828996.514773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787828996.514777:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333237632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828996.514788:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828996.514789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828996.514795:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828996.514798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb221c00 x1874663333237248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828996.514799:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514809:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00100000:2.0:1787828996.514816:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333237248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828996.514816:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514825:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787828996.514827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828996.514846:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828996.514849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.514854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828996.514860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787828996.514863:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333237760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828996.514876:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828996.514881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787828996.514881:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828996.514886:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828996.514892:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10807 00000100:00080000:3.0:1787828996.514896:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828996.514900:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787828996.514906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb222d80 x1874663333237376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828996.514908:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828996.514922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333237376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828996.514928:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333237888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828996.514934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.514942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.514945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828996.514960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828996.514974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828996.514982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb223100 x1874663333237504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828996.514989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333237504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828996.514994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.514996:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.514998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333237504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787828996.515001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828996.515011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828996.515017:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb221880 x1874663333237632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828996.515024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333237632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828996.515034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.515036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.515038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828996.515041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787828996.515052:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828996.515057:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb220700 x1874663333237760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828996.515064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333237760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828996.515069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.515071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.515074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787828996.515077:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787828996.515086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787828996.515091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828996/real 1787828996] req@ffff9057fb222a00 x1874663333237888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828996.515098:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333237888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828996.515108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828996.515110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787828996.515112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333237888:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787828996.898525:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828997.922355:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828998.946427:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828999.970330:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829000.994392:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829001.634371:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634382:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829001.634387:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634397:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634409:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829001.634456:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333238016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.634489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634492:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829001.634493:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634499:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634502:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.634521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333238144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.634528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.634533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634536:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634537:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.634550:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333238272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.634558:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829001.634562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634564:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634574:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829001.634581:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333238016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829001.634586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333238400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.634605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.634609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634613:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634615:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634617:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829001.634630:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333238528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.634636:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829001.634641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634644:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634646:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634650:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634653:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.634663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333238656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829001.634669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829001.634687:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634691:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.634693:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634697:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634703:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.634721:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333238784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829001.634723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158dc00 x1874663333238016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829001.634747:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.634757:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.634763:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.634775:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10812 00000100:00080000:0.0:1787829001.634778:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.634780:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829001.634784:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829001.634791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333238912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829001.634814:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333238016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.634831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.634840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.634845:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333238016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829001.634854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.634876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.634888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158e300 x1874663333238144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.634900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333238144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.634908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.634911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.634914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.634919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.634934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.634942:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158f100 x1874663333238272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.634952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333238272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.634959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.634962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.634965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.634989:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829001.635006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.635015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158ca80 x1874663333238400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.635025:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333238400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.635033:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.635035:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.635038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829001.635044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829001.635057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.635065:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158df80 x1874663333238528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.635075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333238528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.635082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.635085:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.635087:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333238528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829001.635092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.635106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.635114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158e680 x1874663333238656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.635131:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333238656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.635138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.635140:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.635143:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.635148:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.635162:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.635169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158f800 x1874663333238784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.635179:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333238784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.635186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.635189:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.635191:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.635196:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829001.635210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829001.635218:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff9057f158c000 x1874663333238912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.635269:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333238912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.635285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.635288:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829001.635292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333238912:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829002.018402:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829003.042413:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829004.066440:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829005.090402:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829006.114364:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829006.754334:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829006.754350:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754370:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754382:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754389:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829006.754427:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333239040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829006.754453:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754458:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829006.754460:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754465:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754469:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754474:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754484:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829006.754499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333239040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829006.754501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333239168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829006.754519:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829006.754524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754529:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754533:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754538:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754543:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829006.754557:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333239296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829006.754567:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829006.754571:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829006.754573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829006.754577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754582:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754587:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754592:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829006.754611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333239424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829006.754618:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78aaa00 x1874663333239040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829006.754623:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829006.754637:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754644:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754648:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829006.754650:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333239040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829006.754655:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829006.754670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.754677:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.754684:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333239040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829006.754685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333239552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829006.754698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754701:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829006.754704:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829006.754707:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239168:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829006.754712:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754718:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754723:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829006.754734:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829006.754741:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333239680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829006.754748:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78aa680 x1874663333239168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829006.754755:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754775:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829006.754778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333239168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829006.754779:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754788:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754795:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829006.754796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829006.754803:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829006.754803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.754812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239168:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829006.754813:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829006.754821:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829006.754826:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333239808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829006.754835:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829006.754838:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829006.754839:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829006.754842:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10817 00000100:00080000:1.0:1787829006.754845:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829006.754849:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829006.754852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829006.754854:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829006.754864:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333239936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829006.754872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78a9c00 x1874663333239296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.754889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333239296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.754898:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.754901:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.754905:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829006.754912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829006.754932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829006.754943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78aa300 x1874663333239424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.754953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333239424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.754959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.754961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.754963:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829006.754968:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829006.754983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829006.754995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78a9180 x1874663333239552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.755007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333239552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.755024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.755028:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.755031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333239552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829006.755047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829006.755075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829006.755089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78a8700 x1874663333239680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.755102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333239680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.755110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.755114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.755117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829006.755123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829006.755143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829006.755154:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78a9500 x1874663333239808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.755166:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333239808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.755175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.755178:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.755182:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829006.755188:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829006.755207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829006.755218:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff9057c78ab100 x1874663333239936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.755264:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333239936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.755273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.755276:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829006.755280:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333239936:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829007.138389:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829008.162388:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829009.186372:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829010.210361:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829011.234462:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829011.874381:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874390:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829011.874395:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874414:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874424:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874431:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829011.874467:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333240064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874504:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829011.874509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874513:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874516:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874519:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874526:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829011.874536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333240192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874544:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.874548:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874550:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874552:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874555:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829011.874564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333240320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874572:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829011.874576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874580:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874584:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874587:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874591:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829011.874599:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333240064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829011.874601:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333240448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874616:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874618:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.874620:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874623:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874626:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874628:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874632:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829011.874645:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333240576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874667:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829011.874669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874673:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874676:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874678:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874682:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829011.874687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829011.874694:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333240704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874711:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.874712:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874716:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00080000:2.0:1787829011.874719:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874746:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874751:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829011.874758:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de0a80 x1874663333240064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829011.874764:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333240832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.874803:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.874806:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.874808:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.874815:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10822 00000100:00100000:1.0:1787829011.874816:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333240064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829011.874818:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.874825:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829011.874833:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829011.874842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829011.874848:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333240960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829011.874852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.874862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333240064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829011.874873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.874896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.874907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de0000 x1874663333240192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.874916:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333240192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.874923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.874925:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.874928:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.874932:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.874945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.874952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de1c00 x1874663333240320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.874961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333240320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.874967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.874969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.874972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.874992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829011.875006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.875014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de1500 x1874663333240448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.875023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333240448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.875029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.875031:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.875033:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829011.875037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829011.875050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.875056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de0700 x1874663333240576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.875065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333240576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.875071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.875073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.875075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333240576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829011.875079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.875091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.875098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de3800 x1874663333240704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.875112:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333240704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.875118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.875120:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.875122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.875126:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.875138:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.875145:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de1880 x1874663333240832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.875153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333240832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.875159:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.875161:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.875163:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.875167:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829011.875179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829011.875185:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff9057d0de1180 x1874663333240960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.875193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333240960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.875205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.875207:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829011.875210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333240960:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829012.258382:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829013.282396:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829014.306364:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829015.330440:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829016.354381:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829016.994328:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994337:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829016.994342:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994352:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994379:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994389:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829016.994434:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333241088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829016.994463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829016.994469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994474:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994495:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994499:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829016.994508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829016.994522:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333241216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829016.994534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829016.994542:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333241088:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829016.994544:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994557:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994563:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994569:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829016.994577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829016.994599:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333241344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829016.994615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829016.994616:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829016.994623:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829016.994626:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994633:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994641:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994645:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829016.994652:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829016.994664:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb220000 x1874663333241088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829016.994670:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333241472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829016.994730:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829016.994735:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333241088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829016.994746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829016.994751:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994759:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:1.0:1787829016.994760:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829016.994762:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829016.994767:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829016.994770:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829016.994790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333241088:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829016.994796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829016.994813:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333241600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829016.994822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241216:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829016.994830:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829016.994838:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.994844:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994849:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994854:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829016.994861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829016.994863:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829016.994879:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333241728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829016.994883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb220a80 x1874663333241216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829016.994908:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994915:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829016.994917:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829016.994924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333241216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829016.994925:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.994932:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.994940:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829016.994949:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829016.994959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829016.994964:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333241856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829016.994972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.994985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241216:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829016.994986:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829016.994998:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829016.995002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241344:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829016.995002:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829016.995042:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10827 00000100:00080000:3.0:1787829016.995048:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829016.995053:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829016.995058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829016.995059:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829016.995069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb221180 x1874663333241344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829016.995077:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333241984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829016.995083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333241344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995103:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829016.995111:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829016.995135:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829016.995142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb222680 x1874663333241472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829016.995149:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333241472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829016.995168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829016.995179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829016.995184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb220e00 x1874663333241600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829016.995191:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333241600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333241600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829016.995203:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829016.995216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829016.995226:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb221500 x1874663333241728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829016.995237:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333241728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995249:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995253:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829016.995259:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829016.995302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829016.995311:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb223800 x1874663333241856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829016.995330:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333241856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995361:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995366:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829016.995374:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829016.995402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829016.995414:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829016] req@ffff9057fb223b80 x1874663333241984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829016.995425:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333241984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829016.995432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829016.995435:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829016.995438:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333241984:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829017.378393:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829018.402484:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829019.426401:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829020.450391:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829021.474415:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829022.114333:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829022.114345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114353:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114362:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114369:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829022.114402:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333242112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114444:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829022.114445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114450:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114453:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829022.114471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333242240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114479:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114482:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829022.114483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114485:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114487:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114500:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829022.114509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333242112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829022.114514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333242368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114546:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829022.114549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114552:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114557:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114561:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829022.114591:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333242496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114602:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829022.114608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114612:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114616:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829022.114618:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829022.114623:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114628:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829022.114640:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333242624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114647:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114649:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829022.114663:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114684:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114688:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114692:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114696:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829022.114701:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78a8a80 x1874663333242112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829022.114710:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333242752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829022.114727:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829022.114729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333242112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829022.114730:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829022.114733:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114741:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114746:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114752:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829022.114754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829022.114755:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829022.114764:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829022.114768:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333242880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829022.114773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333242112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829022.114782:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829022.114785:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829022.114787:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242240:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829022.114788:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829022.114796:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10832 00000100:00080000:1.0:1787829022.114798:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829022.114800:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829022.114804:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829022.114823:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333243008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829022.114833:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.114849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78ab100 x1874663333242240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.114863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333242240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.114872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.114875:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.114879:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.114901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829022.114917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.114926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78a9500 x1874663333242368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.114957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333242368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.114965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.114968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.114972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.114978:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829022.114993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.115001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78a8700 x1874663333242496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.115013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333242496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.115021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.115024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.115027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829022.115033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829022.115047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.115056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78a9180 x1874663333242624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.115067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333242624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.115083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.115086:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.115090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333242624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829022.115095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333242752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829022.115124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.115133:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78aa300 x1874663333242752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.115152:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333242752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.115161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.115164:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.115167:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333242752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.115182:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333242880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829022.115199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.115209:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78a9c00 x1874663333242880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.115266:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333242880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.115274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.115277:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.115281:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333242880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.115303:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829022.115324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829022.115336:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9057c78aa680 x1874663333243008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.115348:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333243008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.115357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.115361:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829022.115364:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243008:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829022.498370:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829023.522408:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829024.546369:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829025.570469:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829026.594375:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829027.234287:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234296:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829027.234300:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234308:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234317:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829027.234323:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829027.234352:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333243136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829027.234375:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234378:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829027.234379:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234383:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234385:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829027.234388:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234396:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829027.234405:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333243264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829027.234413:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234417:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829027.234420:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829027.234420:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333243136:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829027.234428:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234431:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829027.234433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234437:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829027.234448:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333243392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829027.234459:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234461:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829027.234463:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234465:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234468:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829027.234478:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829027.234483:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829027.234501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333243520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829027.234512:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829027.234516:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78aaa00 x1874663333243136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829027.234518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829027.234523:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234533:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234537:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829027.234541:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333243136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829027.234545:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829027.234557:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234562:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829027.234568:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333243648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829027.234569:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333243136:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829027.234582:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829027.234587:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234591:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234593:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829027.234595:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243264:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829027.234599:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234609:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829027.234636:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333243776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829027.234644:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829027.234645:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829027.234650:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829027.234651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234655:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234659:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829027.234662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78ab480 x1874663333243264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829027.234665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234673:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829027.234679:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333243264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829027.234684:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333243904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829027.234689:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234696:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234699:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243264:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829027.234705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829027.234707:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243392:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829027.234713:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829027.234716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829027.234720:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10837 00000100:00080000:1.0:1787829027.234723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829027.234726:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829027.234730:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829027.234730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.234746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78abb80 x1874663333243392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829027.234747:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333244032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829027.234783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333243392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.234790:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234792:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234795:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829027.234802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829027.234814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.234820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78a8000 x1874663333243520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829027.234832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333243520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.234837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829027.234843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829027.234853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.234858:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78aad80 x1874663333243648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829027.234864:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333243648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.234869:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234873:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333243648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829027.234883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829027.234893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.234899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78a9880 x1874663333243776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829027.234906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333243776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.234915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234919:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829027.234922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829027.234937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.234947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78ab800 x1874663333243904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829027.234960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333243904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.234969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.234972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.234975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333243904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829027.234982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829027.234997:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829027.235004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829027/real 1787829027] req@ffff9057c78a8380 x1874663333244032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829027.235011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333244032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829027.235017:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829027.235019:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829027.235021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244032:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829027.618396:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829028.642421:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829029.666430:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829030.690376:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829031.714381:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829032.354353:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354361:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829032.354365:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354372:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354381:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354388:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829032.354419:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333244160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829032.354441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354444:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829032.354445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354450:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354453:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829032.354470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333244288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829032.354478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354483:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829032.354484:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333244160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829032.354485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354488:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354490:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354495:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354500:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829032.354508:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333244416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829032.354514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354516:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829032.354517:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354520:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354522:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354524:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829032.354534:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333244544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829032.354540:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354542:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829032.354543:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354546:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354549:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829032.354550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829032.354553:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829032.354566:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333244672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829032.354573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829032.354577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354599:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354602:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829032.354603:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78a9f80 x1874663333244160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829032.354607:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829032.354627:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333244160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829032.354631:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333244800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829032.354648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829032.354652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829032.354655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829032.354657:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829032.354660:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829032.354661:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333244160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829032.354667:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354671:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354673:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829032.354684:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333244928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829032.354688:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244288:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829032.354701:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829032.354704:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829032.354705:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829032.354709:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10843 00000100:00080000:1.0:1787829032.354712:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829032.354714:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829032.354717:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829032.354718:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829032.354730:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333245056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829032.354734:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78a8380 x1874663333244288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.354757:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333244288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.354769:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.354773:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.354778:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829032.354783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829032.354796:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.354803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78ab800 x1874663333244416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.354811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333244416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.354816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.354818:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.354820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829032.354824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829032.354834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.354839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78a9880 x1874663333244544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.354849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333244544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.354858:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.354861:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.354864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829032.354871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829032.354891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.354903:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78aad80 x1874663333244672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.354918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333244672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.354923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.354925:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.354927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333244672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829032.354931:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829032.354945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.354951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78a8000 x1874663333244800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.354959:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333244800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.354964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.354965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.354967:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829032.354980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829032.354990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.354996:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78abb80 x1874663333244928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.355003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333244928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.355013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.355015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.355017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333244928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829032.355020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829032.355030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829032.355035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829032/real 1787829032] req@ffff9057c78ab480 x1874663333245056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829032.355042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333245056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829032.355047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829032.355049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829032.355050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245056:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829032.738490:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829033.762367:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829034.786378:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829035.810491:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829036.834420:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829037.474299:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474311:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829037.474315:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474325:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474336:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829037.474351:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829037.474391:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333245184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829037.474421:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474425:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829037.474428:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474432:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474437:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829037.474442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474452:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829037.474467:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333245184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829037.474469:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333245312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829037.474487:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474491:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829037.474493:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474498:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474502:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829037.474507:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474513:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829037.474527:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333245440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829037.474539:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829037.474557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829037.474559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829037.474563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474569:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474574:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829037.474579:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829037.474600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333245568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829037.474604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78aaa00 x1874663333245184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829037.474621:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474626:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829037.474628:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474632:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474635:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829037.474637:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333245184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829037.474641:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829037.474657:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.474664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829037.474668:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333245696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829037.474673:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333245184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829037.474689:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829037.474695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474701:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474705:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829037.474706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245312:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829037.474713:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829037.474734:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333245824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829037.474743:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829037.474743:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829037.474748:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829037.474750:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829037.474754:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:1.0:1787829037.474758:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829037.474765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78aa680 x1874663333245312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829037.474769:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474776:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829037.474789:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333245952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829037.474794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333245312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829037.474801:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829037.474809:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829037.474812:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829037.474816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829037.474824:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10848 00000100:00080000:2.0:1787829037.474826:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829037.474829:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829037.474833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245312:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829037.474835:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829037.474843:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829037.474849:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829037.474853:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333246080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829037.474880:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.474894:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78a9c00 x1874663333245440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.474908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333245440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.474917:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.474921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.474925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.474932:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829037.474953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.474966:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78aa300 x1874663333245568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.474980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333245568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.474989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.474993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.474996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829037.475003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829037.475024:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.475036:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78a9180 x1874663333245696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.475049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333245696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.475058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.475061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.475065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333245696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829037.475083:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829037.475104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.475116:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78a8700 x1874663333245824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.475129:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333245824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.475138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.475141:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.475144:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.475151:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829037.475170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.475181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78a9500 x1874663333245952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.475193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333245952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.475202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.475206:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.475209:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333245952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.475216:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829037.475264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829037.475277:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829037/real 1787829037] req@ffff9057c78ab100 x1874663333246080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.475291:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333246080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.475300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.475303:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829037.475306:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246080:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829037.858467:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829038.882393:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829039.906472:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829040.930352:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829041.954457:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829042.594318:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594328:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829042.594333:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594343:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594355:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594361:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829042.594393:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333246208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594418:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594421:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829042.594422:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594425:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594428:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594431:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594438:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829042.594448:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333246336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829042.594455:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333246208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829042.594458:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594462:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829042.594464:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594468:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594471:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594474:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829042.594482:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333246464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594491:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829042.594493:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594495:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594506:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594521:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594526:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829042.594535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333246592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594545:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829042.594546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594549:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594553:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594556:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829042.594560:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246336:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787829042.594562:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829042.594575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333246720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594584:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594587:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829042.594594:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246464:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829042.594595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594603:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594607:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829042.594611:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829042.594620:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829042.594626:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333246848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829042.594642:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246720:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829042.594650:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829042.594654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594660:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594665:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829042.594666:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829042.594673:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594680:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829042.594695:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333246976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829042.594705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829042.594707:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829042.594709:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829042.594713:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10853 00000100:00080000:3.0:1787829042.594716:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829042.594718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246976:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829042.594722:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829042.594728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829042.594739:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333247104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829042.594754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333247104:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829042.978356:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829044.002424:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829045.026363:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829045.666485:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829045.666551:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829045.666566:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666602:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666608:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666614:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666618:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666623:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666627:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829045.666633:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829045.666676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb220000 x1874663333247104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666702:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333247104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666717:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333247104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829045.666730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb220a80 x1874663333246976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666745:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333246976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666756:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.666766:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb221180 x1874663333246848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666777:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333246848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666786:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666790:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666793:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333246848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.666800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb222680 x1874663333246720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333246720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829045.666843:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb223800 x1874663333246336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666854:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333246336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666865:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.666874:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb221500 x1874663333246464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333246464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666897:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666901:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829045.666906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb220e00 x1874663333246592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333246592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333246592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829045.666938:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829045] req@ffff9057fb223b80 x1874663333246208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829045.666951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333246208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829045.666959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829045.666963:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829045.666966:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333246208:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1787829046.050416:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829047.074456:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829047.714352:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829047.714370:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714384:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714398:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714407:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829047.714453:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333247232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714501:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714505:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829047.714507:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714512:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714516:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714520:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714533:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829047.714548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333247360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714561:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829047.714569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829047.714571:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333247232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829047.714575:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714583:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714587:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714592:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829047.714606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333247488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714616:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829047.714623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714628:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714652:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714656:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714662:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829047.714669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829047.714675:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333247616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714688:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714692:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829047.714694:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714699:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714703:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714706:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829047.714729:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb222a00 x1874663333247232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829047.714730:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333247744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714756:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714760:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829047.714762:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714769:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00100000:2.0:1787829047.714770:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333247232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829047.714772:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714781:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714790:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829047.714793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.714801:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829047.714804:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333247872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829047.714810:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333247232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829047.714824:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714829:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829047.714831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829047.714835:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714838:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829047.714843:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714848:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829047.714852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829047.714862:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333248000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829047.714876:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829047.714881:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829047.714884:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829047.714886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829047.714891:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10858 00000100:00080000:3.0:1787829047.714908:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829047.714914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb220700 x1874663333247360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829047.714916:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829047.714925:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829047.714939:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333247360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829047.714943:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333248128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829047.714974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.714979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.714983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829047.714992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829047.715021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb221880 x1874663333247488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333247488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829047.715070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829047.715098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb223100 x1874663333247616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715125:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333247616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715137:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829047.715148:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829047.715164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715173:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb222d80 x1874663333247744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715185:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333247744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715202:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333247744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829047.715208:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333247872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829047.715267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715281:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb221c00 x1874663333247872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715294:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333247872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715306:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715309:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333247872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829047.715336:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829047.715359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715371:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb222300 x1874663333248000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715382:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333248000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715392:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715395:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829047.715401:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829047.715437:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829047.715450:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff9057fb220380 x1874663333248128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.715462:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333248128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.715471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.715486:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829047.715491:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248128:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829048.098492:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829049.122413:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829050.146370:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829051.170335:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829052.194377:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829052.834306:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834316:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829052.834321:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829052.834332:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834341:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829052.834348:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829052.834387:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333248256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829052.834414:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834418:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829052.834419:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829052.834424:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834428:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829052.834432:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829052.834470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333248384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829052.834475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333248256:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829052.834491:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829052.834497:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829052.834502:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834506:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829052.834511:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834516:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829052.834530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333248512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829052.834538:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829052.834553:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834557:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829052.834559:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829052.834564:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834569:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829052.834573:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb220380 x1874663333248256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829052.834576:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829052.834609:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333248256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829052.834610:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333248640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829052.834629:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.834635:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829052.834636:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829052.834639:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333248256:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829052.834645:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829052.834648:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829052.834653:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834657:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829052.834662:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829052.834663:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834672:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829052.834687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829052.834689:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333248768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829052.834699:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb222300 x1874663333248384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829052.834705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834712:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829052.834713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333248384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829052.834716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829052.834726:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829052.834727:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:2.0:1787829052.834732:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.834736:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829052.834736:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829052.834747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248512:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829052.834748:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834758:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829052.834764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829052.834777:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb221c00 x1874663333248512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829052.834778:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333248896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829052.834794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333248512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829052.834804:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834815:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829052.834816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829052.834819:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829052.834820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.834826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248512:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829052.834830:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00100000:2.0:1787829052.834835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248640:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829052.834839:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829052.834846:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829052.834852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834853:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829052.834860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb222d80 x1874663333248640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829052.834872:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333248640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829052.834872:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333249024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829052.834895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.834900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.834905:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248640:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829052.834906:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829052.834912:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829052.834914:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829052.834918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248768:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829052.834921:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10863 00000100:00080000:3.0:1787829052.834928:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829052.834935:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829052.834936:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829052.834945:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829052.834947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb223100 x1874663333248768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829052.834957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333248768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829052.834963:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333249152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829052.834965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.834971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.834978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333248768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829052.834989:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829052.835008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829052.835016:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb221880 x1874663333248896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829052.835025:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333248896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829052.835030:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.835032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.835034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333248896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829052.835037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829052.835047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829052.835053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb220700 x1874663333249024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829052.835059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333249024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829052.835064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.835065:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.835067:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829052.835075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829052.835085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829052.835092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff9057fb222a00 x1874663333249152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829052.835099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333249152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829052.835104:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829052.835105:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829052.835107:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249152:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829053.218348:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829054.242465:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829055.266413:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829056.290366:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829057.314390:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829057.954313:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954321:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829057.954324:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954332:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954342:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829057.954348:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829057.954381:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333249280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829057.954420:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954425:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829057.954427:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954432:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954437:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829057.954442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829057.954458:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333249280:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829057.954458:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829057.954475:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333249408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829057.954484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829057.954488:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954491:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954507:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829057.954510:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829057.954515:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829057.954516:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829057.954528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333249536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829057.954536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954539:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829057.954540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954543:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954546:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829057.954549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829057.954554:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb223b80 x1874663333249280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829057.954555:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829057.954568:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333249664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829057.954574:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333249280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829057.954578:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829057.954588:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829057.954596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829057.954596:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954599:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829057.954602:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.954605:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333249280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829057.954610:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829057.954615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829057.954623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333249792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829057.954629:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249408:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829057.954633:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829057.954638:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954643:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829057.954651:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829057.954654:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829057.954660:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829057.954666:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb220e00 x1874663333249408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829057.954670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333249920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829057.954679:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954682:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829057.954684:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954690:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954694:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829057.954699:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333249408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829057.954700:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829057.954709:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829057.954714:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.954721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829057.954725:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333250048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829057.954730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829057.954744:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249536:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829057.954746:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829057.954753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829057.954755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829057.954761:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10868 00000100:00080000:3.0:1787829057.954769:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829057.954773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829057.954778:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829057.954785:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829057.954786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb221500 x1874663333249536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.954799:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333249536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829057.954802:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333250176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829057.954819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.954825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.954829:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829057.954834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829057.954849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829057.954856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb223800 x1874663333249664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.954863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333249664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829057.954868:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.954870:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.954872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829057.954883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829057.954893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829057.954899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb222680 x1874663333249792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.954906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333249792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829057.954911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.954913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.954915:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333249792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829057.954919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829057.954932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829057.954943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb221180 x1874663333249920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.954956:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333249920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829057.954965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.954968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.954972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333249920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829057.954978:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829057.954993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829057.955001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb220a80 x1874663333250048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.955010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333250048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829057.955019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.955022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.955025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829057.955031:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829057.955051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829057.955063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff9057fb220000 x1874663333250176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829057.955071:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333250176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829057.955077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829057.955079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829057.955081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250176:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829058.338429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829059.362335:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829060.386347:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829061.410532:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829062.434387:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829063.074315:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074325:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829063.074330:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074341:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074359:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829063.074415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333250304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074442:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074445:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.074447:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074450:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074453:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074456:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074463:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.074474:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333250432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074486:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.074488:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074491:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074493:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074496:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074499:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.074506:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333250560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.074518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074521:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074522:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074526:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074534:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829063.074536:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333250304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829063.074546:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333250688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074555:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074557:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.074558:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074560:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074563:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074566:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074570:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829063.074578:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333250816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074584:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074586:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.074587:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074591:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074593:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074596:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.074607:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333250944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074612:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074614:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.074615:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074619:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074622:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074627:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829063.074638:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829063.074639:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333251072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.074662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.074666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.074669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.074674:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10873 00000100:00080000:1.0:1787829063.074679:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.074685:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829063.074693:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829063.074704:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78a8a80 x1874663333250304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829063.074711:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333251200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829063.074795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333250304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.074809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.074815:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.074819:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333250304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829063.074829:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.074855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.074868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78ab100 x1874663333250432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.074882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333250432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.074892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.074896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.074900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.074928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.074950:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.074962:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78a9500 x1874663333250560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.074976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333250560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.074993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.074997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075001:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.075008:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.075031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.075043:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78a8700 x1874663333250688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.075056:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333250688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.075066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.075070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829063.075081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.075099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.075109:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78a9180 x1874663333250816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.075122:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333250816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.075132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.075136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075140:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333250816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829063.075147:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.075164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.075175:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78aa300 x1874663333250944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.075187:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333250944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.075202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.075206:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333250944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.075217:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.075284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.075297:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78a9c00 x1874663333251072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.075311:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333251072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.075320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.075324:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075327:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.075334:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.075354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829063.075381:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9057c78aa680 x1874663333251200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.075395:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333251200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.075404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.075407:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829063.075411:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251200:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829063.458440:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829064.482467:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829065.506465:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829066.530459:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829067.554412:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829068.194346:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194354:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829068.194357:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194386:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194393:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829068.194434:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333251328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.194454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194457:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.194458:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194462:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194464:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194466:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829068.194481:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333251456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.194489:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194492:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.194493:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194496:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194498:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194502:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194507:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829068.194512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333251328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829068.194519:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333251584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.194528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.194532:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194535:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194537:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194540:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829068.194552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333251712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.194559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.194563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194568:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194572:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829068.194574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829068.194587:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333251840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.194594:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.194598:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194602:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:3.0:1787829068.194606:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194609:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829068.194613:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb220000 x1874663333251328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829068.194614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829068.194628:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333251968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.194636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333251328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829068.194638:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.194651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10878 00000100:00080000:2.0:1787829068.194659:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829068.194661:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829068.194667:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829068.194668:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194675:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829068.194676:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333251328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829068.194684:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333252096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.194687:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251456:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829068.194702:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.194705:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.194707:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.194710:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10878 00000800:00000100:2.0:1787829068.194715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829068.194715:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.194723:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829068.194727:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829068.194731:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb220a80 x1874663333251456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829068.194739:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333252224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.194747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333251456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.194755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.194758:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.194760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.194805:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829068.194819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.194825:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb221180 x1874663333251584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.194832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333251584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.194837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.194839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.194841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.194844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.194855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.194861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb222680 x1874663333251712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.194885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333251712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.194891:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.194892:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.194895:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829068.194899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.194922:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.194931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb223800 x1874663333251840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.194939:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333251840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.194945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.194947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.194949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333251840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829068.194953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829068.194970:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.194977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb221500 x1874663333251968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.194985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333251968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.194991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.194993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.194995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333251968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.194999:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829068.195011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.195018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb220e00 x1874663333252096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.195026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333252096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.195031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.195033:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.195035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.195039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.195050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829068.195056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9057fb223b80 x1874663333252224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.195063:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333252224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.195068:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.195071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829068.195073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252224:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829068.578517:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829069.602400:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829070.626380:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829071.650362:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829072.674413:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829073.314357:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314368:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829073.314373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314383:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314395:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314402:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829073.314442:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333252352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829073.314470:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829073.314475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829073.314511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333252480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829073.314535:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333252352:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829073.314550:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829073.314557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314562:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314567:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314571:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829073.314589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333252608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829073.314598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829073.314604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314610:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10884 00000800:00000100:2.0:1787829073.314610:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829073.314617:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314622:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829073.314639:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333252736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829073.314648:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829073.314656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829073.314656:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158c000 x1874663333252352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829073.314665:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314671:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314678:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829073.314703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333252864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829073.314703:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333252352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829073.314727:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314732:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829073.314734:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829073.314734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829073.314742:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:2.0:1787829073.314746:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829073.314749:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829073.314757:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333252352:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829073.314758:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314766:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829073.314779:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333252992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829073.314790:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829073.314794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252480:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829073.314796:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829073.314800:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314808:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314811:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314815:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314820:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829073.314831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333253120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829073.314838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829073.314853:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829073.314858:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829073.314860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158f800 x1874663333252480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829073.314861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829073.314869:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10884 00000100:00080000:0.0:1787829073.314874:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829073.314881:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829073.314886:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829073.314892:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333252480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829073.314899:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333253248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829073.314928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.314935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.314944:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829073.314953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829073.314979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.314993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158e680 x1874663333252608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333252608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315028:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829073.315039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829073.315062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.315074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158df80 x1874663333252736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315087:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333252736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829073.315111:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829073.315133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.315146:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158ca80 x1874663333252864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315160:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333252864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315172:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315177:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333252864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829073.315198:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829073.315248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.315263:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158f100 x1874663333252992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315279:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333252992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315292:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315314:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333252992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829073.315321:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333253120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829073.315344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.315355:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158e300 x1874663333253120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315368:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333253120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315381:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315384:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333253120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829073.315391:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333253248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829073.315411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829073.315424:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829073/real 1787829073] req@ffff9057f158dc00 x1874663333253248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829073.315435:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333253248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829073.315444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829073.315447:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829073.315450:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333253248:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829073.698438:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829074.722434:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829075.746419:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829076.770407:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829077.794362:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829078.434323:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434334:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829078.434338:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829078.434348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434367:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829078.434403:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333253376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829078.434436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829078.434440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434448:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434457:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829078.434470:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333253504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829078.434488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829078.434495:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333253376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829078.434496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434505:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434510:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434515:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829078.434532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333253632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434551:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829078.434554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829078.434559:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434566:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434581:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829078.434582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829078.434607:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333253760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829078.434636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829078.434640:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158d880 x1874663333253376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829078.434644:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434653:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434662:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829078.434687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333253376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829078.434690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333253888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434728:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434736:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829078.434739:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829078.434741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829078.434745:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:0.0:1787829078.434752:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829078.434752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.434765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333253376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829078.434776:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829078.434796:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829078.434804:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333254016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829078.434818:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434822:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829078.434824:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829078.434830:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10889 00000800:00000100:2.0:1787829078.434837:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829078.434840:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829078.434848:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434854:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829078.434861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158ce00 x1874663333253504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829078.434871:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333254144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829078.434886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333253504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829078.434887:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829078.434900:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829078.434904:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829078.434917:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10889 00000100:00080000:2.0:1787829078.434920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829078.434922:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829078.434931:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829078.434932:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829078.434943:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829078.434945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829078.434957:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333254272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829078.434960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829078.435005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435030:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158c700 x1874663333253632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435054:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333253632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829078.435085:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829078.435111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158ea00 x1874663333253760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435138:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333253760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435149:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829078.435158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829078.435173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158d180 x1874663333253888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435192:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333253888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435202:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435205:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333253888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829078.435284:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829078.435310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435322:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158d500 x1874663333254016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435333:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333254016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435345:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435348:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829078.435353:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829078.435368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435377:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158ed80 x1874663333254144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435388:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333254144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435398:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435400:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829078.435405:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829078.435419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829078.435428:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829078/real 1787829078] req@ffff9057f158fb80 x1874663333254272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829078.435439:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333254272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829078.435446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829078.435449:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829078.435451:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333254272:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829078.818470:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829079.842472:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829080.866470:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829081.890425:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829082.914474:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829083.554357:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554365:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829083.554368:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554376:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829083.554423:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333254400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829083.554454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554459:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829083.554461:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554466:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554470:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554475:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829083.554500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333254528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829083.554506:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333254400:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829083.554549:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554554:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829083.554556:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554561:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554576:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554580:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829083.554586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829083.554603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333254656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829083.554612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829083.554617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554621:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554625:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554646:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829083.554648:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb222a00 x1874663333254400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829083.554651:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829083.554664:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333254784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829083.554697:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829083.554702:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554704:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554707:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829083.554709:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333254400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829083.554713:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829083.554727:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829083.554732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333254912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829083.554735:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.554742:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333254400:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829083.554757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554760:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829083.554761:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829083.554765:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254528:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829083.554768:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554772:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554775:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554778:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829083.554788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333255040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829083.554794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829083.554807:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829083.554811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb220700 x1874663333254528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829083.554814:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829083.554819:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554829:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554836:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829083.554841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333254528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829083.554845:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554854:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829083.554864:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333255168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829083.554865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.554873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.554880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254528:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829083.554883:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829083.554886:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829083.554888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254656:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829083.554889:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829083.554898:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10894 00000100:00080000:3.0:1787829083.554902:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829083.554904:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829083.554908:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829083.554916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829083.554920:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333255296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829083.554931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb221880 x1874663333254656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.554941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333254656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.554957:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.554961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.554965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829083.554971:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829083.554993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829083.555002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb223100 x1874663333254784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.555010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333254784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.555016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.555018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.555020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829083.555024:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829083.555037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829083.555043:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb222d80 x1874663333254912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.555050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333254912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.555055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.555056:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.555059:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333254912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829083.555075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829083.555093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829083.555105:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb221c00 x1874663333255040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.555118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333255040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.555124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.555126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.555129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829083.555133:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829083.555147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829083.555154:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb222300 x1874663333255168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.555161:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333255168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.555166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.555168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.555170:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829083.555173:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829083.555183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829083.555189:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829083/real 1787829083] req@ffff9057fb220380 x1874663333255296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829083.555197:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333255296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829083.555202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829083.555203:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829083.555205:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333255296:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829083.938352:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829084.962527:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829085.986380:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829087.010393:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829088.034434:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829088.674381:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.674398:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829088.674402:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829088.674429:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674442:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829088.674452:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829088.674497:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333255424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829088.674547:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.674552:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829088.674554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829088.674560:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674565:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829088.674571:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829088.674598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829088.674608:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333255424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829088.674615:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333255552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829088.674629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.674633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829088.674636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829088.674640:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674652:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829088.674657:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829088.674663:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829088.674665:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829088.674682:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333255680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829088.674694:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.674698:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829088.674700:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829088.674705:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674710:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829088.674710:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78aaa00 x1874663333255424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.674719:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829088.674728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829088.674735:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333255424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829088.674743:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333255808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829088.674750:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.674759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829088.674760:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829088.674766:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333255424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829088.674791:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829088.674792:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829088.674797:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829088.674804:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674809:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829088.674814:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829088.674820:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829088.674826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829088.674837:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333255936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829088.674845:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78ab480 x1874663333255552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.674855:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829088.674861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333255552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.674862:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829088.674867:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829088.674875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.674878:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:2.0:1787829088.674881:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829088.674887:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829088.674887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829088.674897:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829088.674898:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255680:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829088.674907:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829088.674926:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333256064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829088.674936:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829088.674940:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.674945:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829088.674948:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829088.674953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78abb80 x1874663333255680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.674954:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.674961:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829088.674970:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829088.674977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333255680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829088.674978:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829088.674989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.674992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829088.674998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829088.674999:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333256192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829088.675005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255808:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829088.675019:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829088.675023:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829088.675026:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829088.675031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829088.675032:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10899 00000100:00080000:1.0:1787829088.675039:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829088.675044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78a8000 x1874663333255808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.675046:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829088.675055:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829088.675059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333255808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829088.675066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.675071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829088.675073:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333256320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829088.675075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829088.675093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829088.675114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829088.675126:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78aad80 x1874663333255936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829088.675138:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333255936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829088.675146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.675149:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829088.675152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333255936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829088.675158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829088.675177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829088.675201:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78a9880 x1874663333256064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829088.675214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333256064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829088.675248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.675250:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829088.675253:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829088.675257:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829088.675273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829088.675281:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78ab800 x1874663333256192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829088.675289:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333256192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829088.675294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.675296:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829088.675298:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829088.675301:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829088.675312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829088.675317:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff9057c78a8380 x1874663333256320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829088.675324:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333256320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829088.675329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829088.675331:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829088.675333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333256320:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829089.058417:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829090.082403:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829091.106376:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829092.130373:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829093.154382:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829093.794329:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794340:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829093.794345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794355:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794367:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794375:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829093.794438:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333256448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829093.794478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794481:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829093.794483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794486:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794489:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794492:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794502:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829093.794513:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333256576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829093.794520:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333256448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829093.794524:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794529:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829093.794530:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794533:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794538:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829093.794550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333256704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829093.794556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829093.794560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794563:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794566:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794569:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794572:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829093.794579:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333256832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829093.794585:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794588:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829093.794590:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794592:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794595:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794598:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829093.794603:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829093.794622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333256960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829093.794634:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794638:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829093.794641:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829093.794646:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb220380 x1874663333256448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829093.794648:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794656:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794663:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794672:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829093.794700:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333257088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829093.794708:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829093.794709:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333256448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829093.794714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829093.794717:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794725:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794728:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.794731:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829093.794732:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794739:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829093.794743:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829093.794749:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333257216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829093.794753:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333256448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829093.794761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829093.794764:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829093.794766:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829093.794769:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10904 00000100:00080000:3.0:1787829093.794772:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829093.794775:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829093.794778:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829093.794788:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829093.794796:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333257344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829093.794814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.794829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb222300 x1874663333256576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.794838:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333256576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.794844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.794846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.794849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829093.794852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829093.794864:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.794870:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb221c00 x1874663333256704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.794878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333256704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.794888:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.794890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.794893:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829093.794896:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829093.794908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.794914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb222d80 x1874663333256832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.794921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333256832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.794927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.794928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.794930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829093.794933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829093.794943:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.794957:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb223100 x1874663333256960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.794965:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333256960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.794971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.794973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.794975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333256960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829093.794979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829093.794991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.794997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb221880 x1874663333257088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.795005:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333257088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.795009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.795011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.795013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829093.795016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829093.795026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.795032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb220700 x1874663333257216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.795038:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333257216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.795043:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.795045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.795047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829093.795062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829093.795073:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829093.795079:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff9057fb222a00 x1874663333257344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.795087:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333257344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829093.795092:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829093.795094:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829093.795095:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333257344:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829094.178347:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829095.202376:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829096.226391:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829097.250546:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829098.274453:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829098.914378:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829098.914393:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914404:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914416:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914433:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829098.914478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333257472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.914505:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829098.914509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914513:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914515:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914518:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829098.914537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333257600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.914545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.914549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914554:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914558:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829098.914558:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333257472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829098.914565:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914569:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829098.914577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333257728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.914589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829098.914594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914596:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914599:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914601:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829098.914612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333257856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.914618:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914620:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.914621:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914631:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10909 00000800:00000100:2.0:1787829098.914636:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829098.914638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914643:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829098.914657:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333257984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.914664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914667:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829098.914668:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914671:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914675:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914678:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829098.914690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333258112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829098.914694:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158fb80 x1874663333257472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829098.914702:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914711:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.914714:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914719:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914723:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829098.914724:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333257472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829098.914729:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914736:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829098.914741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829098.914746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333258240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829098.914749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.914758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333257472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829098.914762:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.914765:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.914767:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.914770:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10909 00000100:00080000:0.0:1787829098.914773:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.914775:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829098.914777:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829098.914787:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829098.914788:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333258368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829098.914819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.914837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158ed80 x1874663333257600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.914851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333257600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.914861:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.914865:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.914867:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829098.914872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829098.914888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.914896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158d500 x1874663333257728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.914904:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333257728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.914909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.914911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.914914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829098.914917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829098.914929:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.914934:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158d180 x1874663333257856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.914941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333257856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.914946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.914948:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.914951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829098.914955:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829098.914965:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.914970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158ea00 x1874663333257984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.914977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333257984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.914983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.914984:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.914986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333257984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829098.914989:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829098.914999:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.915005:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158c700 x1874663333258112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.915011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333258112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.915016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.915018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.915020:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829098.915023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829098.915033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.915039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158ce00 x1874663333258240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.915047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333258240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.915052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.915053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.915055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829098.915069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829098.915080:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829098.915087:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff9057f158d880 x1874663333258368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829098.915095:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333258368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829098.915101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829098.915103:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829098.915106:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333258368:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829099.298383:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829100.322453:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829101.346362:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829102.370368:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829103.394396:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829104.034326:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829104.034345:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829104.034349:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829104.034360:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034371:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.034379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829104.034415:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333258496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829104.034444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829104.034448:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829104.034450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829104.034455:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.034464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829104.034474:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829104.034493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333258496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829104.034493:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333258624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829104.034531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829104.034535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829104.034538:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829104.034542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034552:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.034558:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829104.034561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829104.034567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829104.034585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333258752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829104.034595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829104.034598:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb223b80 x1874663333258496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.034601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829104.034603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829104.034609:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034617:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.034626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829104.034634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829104.034636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333258496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.034647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.034652:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829104.034653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333258880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829104.034657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333258496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829104.034672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829104.034676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829104.034678:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829104.034682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258624:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829104.034687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034693:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.034698:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829104.034705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829104.034705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829104.034717:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb220e00 x1874663333258624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.034725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333259008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829104.034731:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333258624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.034751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.034754:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.034755:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.034761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258624:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829104.034763:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829104.034767:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829104.034768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829104.034778:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034785:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829104.034787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829104.034795:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829104.034800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb221500 x1874663333258752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829104.034804:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829104.034811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333258752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.034823:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333259136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.034831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.034838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.034843:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829104.034844:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829104.034848:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258880:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829104.034851:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829104.034855:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829104.034861:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034867:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829104.034871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829104.034874:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829104.034882:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829104.034883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb223800 x1874663333258880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.034896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333258880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.034904:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333259264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.034905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.034911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.034917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333258880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829104.034934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259008:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829104.034938:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829104.034944:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829104.034946:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829104.034951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829104.034956:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10914 00000100:00080000:3.0:1787829104.034961:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829104.034964:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb222680 x1874663333259008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.034970:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829104.034979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333259008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829104.034979:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829104.034991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.034993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.034997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829104.035016:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333259392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829104.035021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829104.035047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829104.035057:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb221180 x1874663333259136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.035067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333259136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.035073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.035075:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.035078:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.035102:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829104.035132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829104.035140:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb220a80 x1874663333259264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.035148:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333259264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.035153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.035155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.035157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.035161:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829104.035172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829104.035178:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9057fb220000 x1874663333259392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.035203:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333259392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.035209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.035211:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829104.035213:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333259392:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829104.418454:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829105.442387:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829106.466382:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829107.490463:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829108.514377:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829109.154288:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154295:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829109.154299:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154307:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154316:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154323:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829109.154359:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333259520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154388:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154390:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829109.154392:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154396:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154401:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154406:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829109.154415:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829109.154430:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333259648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154442:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829109.154449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154452:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154455:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829109.154462:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333259520:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829109.154465:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829109.154477:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333259776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829109.154489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154492:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154509:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154512:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829109.154516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829109.154523:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333259904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154530:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829109.154534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154536:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154539:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829109.154546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829109.154547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.154558:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333260032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154568:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829109.154570:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154573:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154575:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154578:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829109.154582:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259776:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787829109.154584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829109.154601:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333260160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829109.154606:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259904:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829109.154623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154627:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829109.154628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260032:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829109.154631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154638:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154645:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829109.154648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260160:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829109.154650:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829109.154658:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829109.154672:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333260288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.154683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829109.154685:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829109.154687:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829109.154691:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10919 00000100:00080000:3.0:1787829109.154694:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829109.154696:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829109.154700:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829109.154705:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.154710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333260416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829109.154731:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260416:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829109.538363:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829110.562387:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829111.586410:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829112.226579:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787829112.226626:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829112.226647:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226685:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226693:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226707:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226720:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226731:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829112.226737:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb221180 x1874663333259776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829112.226745:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829112.226757:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829112.226773:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333259776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226793:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829112.226807:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb222680 x1874663333259904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333259904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226831:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829112.226841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb223800 x1874663333260032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333260032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226858:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226862:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829112.226872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb220000 x1874663333259520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333259520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333259520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829112.226906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb220a80 x1874663333259648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333259648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333259648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829112.226956:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb223b80 x1874663333260416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333260416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.226976:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.226979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.226982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829112.226988:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb220e00 x1874663333260288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.226998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333260288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.227006:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.227009:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.227012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829112.227018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829112] req@ffff9057fb221500 x1874663333260160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829112.227048:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333260160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829112.227056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829112.227060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829112.227064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333260160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1787829112.610440:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829113.634404:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829114.274356:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274367:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829114.274371:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829114.274381:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274392:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274401:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829114.274436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333260544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829114.274474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274479:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829114.274481:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829114.274487:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274492:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274497:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829114.274507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829114.274519:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333260544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829114.274525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333260672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829114.274540:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829114.274545:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829114.274550:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274554:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274559:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829114.274564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829114.274579:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829114.274582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333260800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829114.274594:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829114.274605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829114.274610:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274615:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274619:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829114.274626:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829114.274630:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de0e00 x1874663333260544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829114.274642:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333260928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829114.274675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274678:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829114.274683:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829114.274684:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333260544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829114.274692:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829114.274706:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829114.274708:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829114.274716:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.274722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333260544:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787829114.274734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829114.274750:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829114.274755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333261056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829114.274770:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274774:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829114.274777:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829114.274783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829114.274786:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274793:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274798:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829114.274804:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de1f80 x1874663333260672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829114.274805:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829114.274824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333260672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829114.274831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333261184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829114.274837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.274848:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829114.274855:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829114.274856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260672:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829114.274863:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829114.274867:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829114.274869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260800:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829114.274876:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274884:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274888:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829114.274894:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829114.274899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829114.274911:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333261312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829114.274914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de3100 x1874663333260800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829114.274940:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829114.274946:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829114.274947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333260800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829114.274950:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829114.274960:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10925 00000100:00080000:2.0:1787829114.274966:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829114.274974:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829114.274978:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829114.274982:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829114.274986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.274993:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829114.275003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829114.275005:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333261440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829114.275048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829114.275065:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de2a00 x1874663333260928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829114.275077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333260928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829114.275086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.275089:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.275093:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333260928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829114.275111:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829114.275134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829114.275146:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de0e00 x1874663333261056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829114.275158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333261056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829114.275167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.275170:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.275174:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829114.275180:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829114.275203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829114.275216:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de1180 x1874663333261184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829114.275258:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333261184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829114.275266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.275270:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.275273:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829114.275279:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829114.275300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829114.275314:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de1880 x1874663333261312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829114.275327:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333261312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829114.275335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.275339:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.275342:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829114.275349:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829114.275370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829114.275382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829114/real 1787829114] req@ffff9057d0de3800 x1874663333261440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829114.275394:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333261440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829114.275400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829114.275403:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829114.275406:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333261440:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829114.658401:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829115.682394:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829116.706401:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829117.730455:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829118.754449:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829119.394378:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829119.394387:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829119.394391:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394402:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394413:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394421:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829119.394457:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333261568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829119.394487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829119.394492:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829119.394494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394498:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394502:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829119.394516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829119.394531:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333261696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829119.394542:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333261568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829119.394546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829119.394552:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829119.394554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394559:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394564:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394568:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829119.394574:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829119.394589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333261824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829119.394601:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829119.394602:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829119.394608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829119.394611:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394617:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394622:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829119.394632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829119.394638:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158dc00 x1874663333261568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829119.394648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333261952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829119.394655:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333261568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829119.394669:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829119.394672:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829119.394675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829119.394677:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829119.394680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829119.394683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333261568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829119.394689:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394695:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394700:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829119.394704:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261696:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787829119.394708:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829119.394726:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333262080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829119.394731:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829119.394745:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829119.394750:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158e300 x1874663333261696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829119.394751:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829119.394755:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394764:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394771:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394781:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829119.394787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333261696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829119.394788:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829119.394800:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.394804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.394809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829119.394811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333262208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829119.394816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261824:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829119.394832:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829119.394840:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829119.394841:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829119.394846:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829119.394850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158f100 x1874663333261824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829119.394855:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394867:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829119.394869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333261824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829119.394877:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829119.394879:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.394887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:0.0:1787829119.394888:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829119.394894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829119.394904:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333262336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829119.394906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261952:192.168.203.156@tcp:8:kworker.0 00000100:00080000:0.0:1787829119.394922:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829119.394925:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829119.394927:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829119.394930:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10930 00000100:00080000:0.0:1787829119.394934:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829119.394939:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829119.394942:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787829119.394944:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829119.394955:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333262464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829119.394960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158ca80 x1874663333261952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829119.394987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333261952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829119.394998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.395001:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.395004:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333261952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829119.395017:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829119.395032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829119.395039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158df80 x1874663333262080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829119.395046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333262080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829119.395051:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.395053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.395055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829119.395058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829119.395068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829119.395073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158e680 x1874663333262208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829119.395080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333262208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829119.395085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.395086:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.395089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829119.395092:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829119.395102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829119.395107:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158f800 x1874663333262336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829119.395113:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333262336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829119.395118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.395120:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.395122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829119.395125:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829119.395134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829119.395140:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829119/real 1787829119] req@ffff9057f158c000 x1874663333262464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829119.395146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333262464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829119.395150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829119.395152:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829119.395154:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333262464:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829119.778407:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829120.802417:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829121.826416:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829122.850400:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829123.874409:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829124.514358:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514365:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829124.514368:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514376:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514392:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829124.514421:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333262592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829124.514441:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514444:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829124.514445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514450:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514453:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514460:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829124.514468:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333262720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829124.514475:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829124.514479:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514481:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514484:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829124.514488:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333262592:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829124.514490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829124.514505:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333262848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829124.514512:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514521:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829124.514523:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514526:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514529:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514532:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514537:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829124.514547:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333262976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829124.514552:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829124.514557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514561:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829124.514562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514565:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514568:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514573:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829124.514581:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333263104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829124.514587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829124.514593:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de0700 x1874663333262592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829124.514593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514602:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514608:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829124.514619:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333262592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829124.514628:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333263232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829124.514631:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514646:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333262592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829124.514656:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262720:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829124.514662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514667:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829124.514670:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514676:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10935 00000800:00000100:1.0:1787829124.514677:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.514687:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de1500 x1874663333262720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829124.514691:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829124.514703:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829124.514704:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333262720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829124.514713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829124.514715:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514720:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514724:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829124.514734:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333263360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829124.514746:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829124.514749:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829124.514752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829124.514753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829124.514767:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10935 00000100:00080000:2.0:1787829124.514771:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829124.514772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829124.514779:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829124.514785:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829124.514789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de1c00 x1874663333262848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829124.514799:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333263488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829124.514816:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333262848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.514834:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829124.514846:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829124.514863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.514871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de0000 x1874663333262976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829124.514879:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333262976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.514884:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333262976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829124.514892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829124.514903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.514909:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de0a80 x1874663333263104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829124.514916:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333263104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.514928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514931:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829124.514951:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829124.514967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.514975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de3800 x1874663333263232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829124.514982:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333263232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.514988:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.514990:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.514992:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829124.514995:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829124.515005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.515011:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de1880 x1874663333263360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829124.515018:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333263360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.515022:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.515024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.515026:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829124.515038:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829124.515048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829124.515055:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829124/real 1787829124] req@ffff9057d0de1180 x1874663333263488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829124.515062:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333263488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829124.515067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829124.515068:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829124.515070:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333263488:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829124.898382:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829125.922586:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829126.946442:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829127.970414:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829128.994374:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829129.634334:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829129.634345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634355:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634367:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634374:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829129.634412:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333263616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634436:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829129.634437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634441:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634444:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634447:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829129.634464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333263744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829129.634478:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333263616:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829129.634478:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.634483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634490:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634493:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829129.634519:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333263872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829129.634533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634537:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10940 00000800:00000100:1.0:1787829129.634543:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829129.634544:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634554:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829129.634564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333264000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634570:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.634574:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634577:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634580:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634582:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634585:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829129.634587:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb222a00 x1874663333263616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829129.634595:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333264128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634608:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829129.634609:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333263616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829129.634610:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634619:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634623:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829129.634632:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634637:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829129.634640:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333264256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829129.634644:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333263616:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829129.634656:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634658:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.634660:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634663:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10940 00000100:00080000:3.0:1787829129.634665:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634668:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634671:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829129.634672:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829129.634680:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333264384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.634689:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.634691:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.634692:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.634697:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10940 00000800:00000100:1.0:1787829129.634699:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829129.634700:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.634704:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829129.634707:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829129.634715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333264512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829129.634730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb220700 x1874663333263744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634753:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333263744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634760:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.634768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829129.634784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.634792:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb221880 x1874663333263872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634800:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333263872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333263872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.634812:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829129.634823:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.634828:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb223100 x1874663333264000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333264000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634847:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829129.634852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829129.634862:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.634867:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb222d80 x1874663333264128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634874:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333264128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634879:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634881:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634883:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829129.634886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829129.634895:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.634901:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb221c00 x1874663333264256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333264256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634930:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.634948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829129.634962:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.634969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb222300 x1874663333264384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.634976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333264384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.634981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.634983:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.634985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.634988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829129.634998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829129.635004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff9057fb220380 x1874663333264512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.635011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333264512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.635016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.635018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829129.635019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333264512:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829130.018375:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829131.042360:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829132.066421:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829133.090441:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829134.114479:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829134.754347:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754360:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829134.754366:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754379:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754393:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754403:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829134.754456:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333264640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754485:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829134.754489:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754493:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754496:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754500:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754509:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829134.754521:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333264768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829134.754533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829134.754538:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333264640:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829134.754538:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754543:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754548:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754552:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829134.754560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333264896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754574:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829134.754576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754578:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754580:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754583:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829134.754594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333265024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754600:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829134.754604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754607:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754609:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754611:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829134.754619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829134.754623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333265152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754630:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754633:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829134.754635:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754639:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:3.0:1787829134.754645:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754647:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829134.754657:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333265280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829134.754664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829134.754667:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829134.754671:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00100000:1.0:1787829134.754672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb220380 x1874663333264640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829134.754674:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754680:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829134.754686:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829134.754696:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333265408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829134.754700:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333264640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829134.754707:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829134.754714:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829134.754715:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829134.754717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829134.754720:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10945 00000100:00080000:1.0:1787829134.754724:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829134.754724:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829134.754729:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829134.754732:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333264640:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829134.754734:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829134.754745:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333265536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829134.754763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829134.754791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.754806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb222300 x1874663333264768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.754820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333264768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.754830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.754833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.754837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.754844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829134.754866:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.754894:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb221c00 x1874663333264896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.754908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333264896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.754921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.754924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.754926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333264896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.754930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829134.754947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.754954:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb222d80 x1874663333265024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.754962:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333265024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.754967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.754969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.754971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829134.754974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829134.754985:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.754990:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb223100 x1874663333265152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.754997:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333265152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.755003:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.755004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.755007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829134.755011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829134.755021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.755027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb221880 x1874663333265280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.755034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333265280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.755039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.755040:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.755042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.755045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829134.755056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.755062:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb220700 x1874663333265408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.755070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333265408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.755075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.755078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.755080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.755088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829134.755103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829134.755111:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff9057fb222a00 x1874663333265536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.755118:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333265536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.755124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.755126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829134.755128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333265536:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829135.138512:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829136.162405:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829137.186339:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829138.210379:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829139.234388:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829139.874335:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.874345:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829139.874349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00080000:3.0:1787829139.874371:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874378:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829139.874424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333265664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.874451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.874455:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829139.874457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874462:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00080000:3.0:1787829139.874467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874472:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829139.874483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829139.874492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333265664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829139.874500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333265792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.874513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.874518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.874520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874524:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00080000:3.0:1787829139.874529:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874534:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829139.874540:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829139.874547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829139.874565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333265920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.874577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.874582:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829139.874584:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874590:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00100000:1.0:1787829139.874591:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb221500 x1874663333265664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829139.874597:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874605:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829139.874614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829139.874620:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333265664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829139.874629:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333266048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829139.874632:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.874640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829139.874650:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829139.874652:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333265664:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829139.874665:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.874668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874674:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00100000:1.0:1787829139.874676:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265792:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829139.874680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874688:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829139.874694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829139.874708:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333266176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829139.874712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829139.874726:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829139.874731:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb220e00 x1874663333265792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829139.874732:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829139.874737:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.874749:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00080000:3.0:1787829139.874757:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874765:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829139.874770:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333265792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829139.874771:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829139.874783:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.874787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829139.874791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333266304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829139.874792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.874804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265920:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829139.874816:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.874821:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.874823:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829139.874828:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829139.874848:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00100000:1.0:1787829139.874855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb223b80 x1874663333265920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829139.874855:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.874865:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829139.874871:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333265920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829139.874873:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829139.874882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.874885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829139.874887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333265920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.874894:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829139.874894:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333266432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.874912:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829139.874917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829139.874919:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.874923:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829139.874928:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb220a80 x1874663333266048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829139.874929:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10950 00000100:00080000:3.0:1787829139.874938:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829139.874943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333266048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829139.874945:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829139.874955:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829139.874964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.874968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829139.874975:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333266560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829139.874978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829139.875000:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829139.875015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829139.875023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb220000 x1874663333266176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.875031:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333266176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.875036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.875038:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829139.875040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829139.875044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829139.875054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829139.875060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb223800 x1874663333266304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.875067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333266304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.875072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.875074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829139.875076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.875080:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829139.875090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829139.875095:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb222680 x1874663333266432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.875102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333266432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.875107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.875108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829139.875110:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.875114:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829139.875124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829139.875129:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff9057fb221180 x1874663333266560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.875136:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333266560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.875141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.875143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829139.875144:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333266560:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829140.258474:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829141.282384:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829142.306431:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829143.330393:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829144.354385:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829144.994296:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994304:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829144.994307:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994315:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994332:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994338:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829144.994372:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333266688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994393:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994396:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829144.994397:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994400:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994402:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994405:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994412:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829144.994420:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333266816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994427:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994430:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829144.994431:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994435:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994437:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994440:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829144.994446:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333266944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829144.994453:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333266688:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829144.994455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829144.994464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994467:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994469:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994471:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994474:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829144.994482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333267072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994489:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829144.994490:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994493:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994495:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994497:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829144.994507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333267200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994515:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829144.994516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10955 00000800:00000100:1.0:1787829144.994524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829144.994525:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994530:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829144.994539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333267328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829144.994549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994552:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994554:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994557:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829144.994559:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829144.994566:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333267456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829144.994573:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829144.994575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829144.994576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829144.994578:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10955 00000100:00080000:0.0:1787829144.994581:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829144.994587:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994589:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158c000 x1874663333266688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829144.994590:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829144.994601:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333267584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829144.994612:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333266688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994622:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994626:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994629:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333266688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829144.994635:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.994653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158f800 x1874663333266816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994670:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333266816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994675:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994677:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994680:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.994697:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.994709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994715:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158e680 x1874663333266944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994722:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333266944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994732:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994735:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333266944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.994739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829144.994749:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994755:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158df80 x1874663333267072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994761:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333267072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994766:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994768:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994770:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829144.994773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829144.994783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158ca80 x1874663333267200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333267200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994821:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829144.994828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.994843:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994850:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158f100 x1874663333267328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333267328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994864:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994866:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.994869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.994880:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994885:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158e300 x1874663333267456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333267456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994897:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994901:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.994904:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829144.994921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829144.994928:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff9057f158dc00 x1874663333267584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.994939:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333267584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.994944:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.994946:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829144.994948:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333267584:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829145.378505:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829146.402384:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829147.426425:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829148.450333:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829149.474376:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829150.114426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829150.114438:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114448:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114459:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114466:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829150.114508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333267712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829150.114556:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829150.114562:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114567:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114571:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829150.114607:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333267712:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829150.114616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333267840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829150.114626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829150.114631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114633:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114637:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114640:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829150.114653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333267968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829150.114660:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829150.114664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114667:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10960 00000800:00000100:1.0:1787829150.114671:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829150.114672:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114679:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829150.114693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333268096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829150.114701:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114703:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829150.114705:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114708:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114711:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829150.114715:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158d880 x1874663333267712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.114718:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829150.114744:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333268224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829150.114744:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333267712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.114790:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.114791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.114795:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829150.114799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829150.114800:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829150.114806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333267712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829150.114811:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114818:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114824:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114832:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829150.114832:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829150.114854:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333268352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829150.114861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829150.114875:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829150.114876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158ce00 x1874663333267840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.114883:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829150.114887:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114897:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114904:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829150.114910:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829150.114913:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333267840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829150.114917:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829150.114926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.114931:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829150.114935:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333268480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829150.114937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829150.114949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267968:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829150.114952:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829150.114956:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829150.114957:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.114962:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10960 00000100:00080000:0.0:1787829150.114965:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829150.114970:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829150.114971:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829150.114979:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829150.114986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158c700 x1874663333267968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.114991:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333268608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829150.115007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333267968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333267968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829150.115034:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829150.115057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829150.115070:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158ea00 x1874663333268096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829150.115084:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333268096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115105:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115108:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829150.115113:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829150.115130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829150.115138:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158d180 x1874663333268224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829150.115146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333268224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829150.115171:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829150.115183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829150.115190:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158d500 x1874663333268352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829150.115198:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333268352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115205:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115208:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829150.115212:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829150.115249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829150.115258:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158ed80 x1874663333268480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829150.115266:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333268480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115274:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115276:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829150.115280:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829150.115293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829150.115300:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9057f158fb80 x1874663333268608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829150.115308:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333268608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.115313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.115315:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829150.115317:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333268608:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829150.498449:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829151.522404:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829152.546408:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829153.570410:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829154.594463:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829155.234370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234378:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829155.234381:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234389:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234399:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234405:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829155.234439:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333268736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829155.234462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829155.234466:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234469:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234471:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234473:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234480:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829155.234488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333268864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829155.234496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234498:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829155.234499:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234502:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829155.234519:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333268992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829155.234519:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333268736:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829155.234532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829155.234536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234541:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829155.234555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333269120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829155.234561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829155.234565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234567:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234570:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234573:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829155.234586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333269248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829155.234587:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829155.234597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829155.234602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234605:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234607:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234610:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829155.234620:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333269376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829155.234626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829155.234630:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829155.234632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234634:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234637:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829155.234640:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829155.234642:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158fb80 x1874663333268736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829155.234650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333269504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829155.234661:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829155.234663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829155.234666:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829155.234666:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333268736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829155.234675:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10965 00000100:00080000:0.0:1787829155.234681:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829155.234686:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829155.234687:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829155.234690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829155.234694:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333268736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829155.234702:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333269632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829155.234710:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829155.234752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234763:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158ed80 x1874663333268864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234772:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333268864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234777:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234782:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829155.234801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829155.234813:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158d500 x1874663333268992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333268992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234845:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234847:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333268992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829155.234851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829155.234863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158d180 x1874663333269120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333269120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234883:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829155.234888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829155.234899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234904:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158ea00 x1874663333269248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333269248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829155.234923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829155.234934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158c700 x1874663333269376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234946:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333269376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829155.234958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829155.234967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.234972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158ce00 x1874663333269504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.234979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333269504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.234984:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.234986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.234987:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829155.234991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829155.235000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829155.235006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829155/real 1787829155] req@ffff9057f158d880 x1874663333269632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829155.235013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333269632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829155.235018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829155.235020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829155.235021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333269632:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829155.618364:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829156.642379:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829157.666398:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829158.690435:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829159.714409:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829160.354292:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354300:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829160.354303:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354312:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354320:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354326:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829160.354356:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333269760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354379:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354381:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829160.354383:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354385:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354387:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354390:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354397:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829160.354407:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333269888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354416:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829160.354419:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333269760:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829160.354420:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829160.354423:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354430:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354433:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354435:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354438:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829160.354447:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333270016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354456:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829160.354457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354461:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354463:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354465:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354468:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829160.354475:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333270144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354482:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829160.354486:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829160.354487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829160.354490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354495:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354497:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354500:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829160.354510:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333270272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354516:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829160.354520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354523:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354525:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354528:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829160.354538:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78a9f80 x1874663333269760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829160.354540:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333270400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829160.354572:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354574:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829160.354576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354581:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00100000:2.0:1787829160.354581:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333269760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829160.354584:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354590:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829160.354600:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354605:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829160.354607:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333270528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829160.354611:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333269760:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829160.354625:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829160.354627:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829160.354629:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829160.354632:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10971 00000100:00080000:1.0:1787829160.354634:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829160.354636:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829160.354640:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829160.354640:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829160.354653:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333270656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829160.354668:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354681:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78a8380 x1874663333269888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354705:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333269888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333269888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829160.354723:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829160.354739:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78ab800 x1874663333270016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354754:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333270016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354759:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354762:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829160.354766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829160.354778:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78a9880 x1874663333270144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333270144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829160.354803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829160.354813:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78aad80 x1874663333270272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354826:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333270272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829160.354838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829160.354849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78a8000 x1874663333270400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333270400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829160.354873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829160.354883:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78abb80 x1874663333270528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333270528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354915:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829160.354928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829160.354940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829160.354946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829160/real 1787829160] req@ffff9057c78ab480 x1874663333270656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829160.354954:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333270656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829160.354959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829160.354961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829160.354963:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333270656:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829160.738407:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829161.762506:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829162.786383:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829163.810367:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829164.834444:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829165.474376:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829165.474384:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829165.474387:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829165.474395:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474404:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474411:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829165.474442:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333270784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474476:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829165.474478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829165.474480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829165.474483:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474485:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474488:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829165.474507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333270912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474519:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829165.474525:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333270784:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829165.474525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829165.474530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829165.474536:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474540:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474545:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829165.474564:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333271040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474572:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829165.474574:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829165.474576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829165.474580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10976 00000800:00000100:1.0:1787829165.474586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829165.474586:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474594:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829165.474615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333271168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829165.474629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829165.474631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829165.474635:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829165.474644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de0e00 x1874663333270784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829165.474647:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474670:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829165.474690:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333270784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829165.474690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333271296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829165.474717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829165.474717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829165.474719:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829165.474724:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829165.474727:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00100000:1.0:1787829165.474731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333270784:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829165.474732:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474739:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474743:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829165.474756:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829165.474757:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333271424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829165.474774:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829165.474777:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829165.474779:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829165.474783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829165.474786:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474791:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829165.474794:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829165.474798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de2a00 x1874663333270912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829165.474799:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829165.474812:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333271552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829165.474814:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333270912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829165.474834:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829165.474836:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829165.474841:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829165.474843:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829165.474844:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829165.474848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333270912:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829165.474851:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10976 00000100:00080000:2.0:1787829165.474857:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829165.474858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271040:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829165.474863:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829165.474868:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829165.474881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829165.474883:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333271680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829165.474890:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de3100 x1874663333271040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.474906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333271040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.474911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.474914:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.474916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829165.474920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829165.474931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829165.474937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de1f80 x1874663333271168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.474944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333271168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.474953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.474955:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.474957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829165.474960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829165.474969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829165.474975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de1180 x1874663333271296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.474981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333271296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.474986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.474988:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.474990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829165.475009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829165.475020:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829165.475026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de1880 x1874663333271424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.475033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333271424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.475038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.475039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.475042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829165.475045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829165.475055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829165.475061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de3800 x1874663333271552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.475067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333271552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.475072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.475074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.475075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829165.475079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829165.475088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829165.475093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829165/real 1787829165] req@ffff9057d0de0a80 x1874663333271680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829165.475100:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333271680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829165.475105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829165.475106:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829165.475108:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333271680:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829165.858385:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829166.882466:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829167.906433:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829168.930477:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829169.954476:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829170.594326:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829170.594334:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829170.594337:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594345:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594354:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594360:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829170.594391:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333271808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829170.594416:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829170.594418:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829170.594420:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594423:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594425:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594428:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594435:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829170.594443:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333271936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829170.594452:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829170.594453:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333271808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829170.594456:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829170.594460:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594464:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594476:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594478:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829170.594500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333272064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829170.594507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829170.594509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829170.594510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594512:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594515:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594518:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594521:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829170.594526:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829170.594535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333272192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829170.594543:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829170.594546:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829170.594547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594549:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594551:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594554:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829170.594566:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333272320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829170.594571:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78aaa00 x1874663333271808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.594575:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829170.594583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829170.594586:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594592:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594596:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829170.594599:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333271808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.594603:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829170.594615:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829170.594632:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333272448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829170.594635:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829170.594645:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829170.594648:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333271808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829170.594649:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829170.594662:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594669:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594672:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594675:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594678:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829170.594685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333272576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829170.594693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829170.594693:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271936:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829170.594699:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829170.594702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829170.594706:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10981 00000100:00080000:1.0:1787829170.594708:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829170.594711:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829170.594714:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829170.594722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333272704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829170.594732:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.594751:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78aa680 x1874663333271936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.594766:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333271936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.594773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.594776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.594780:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333271936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.594786:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829170.594808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.594821:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78a9c00 x1874663333272064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.594834:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333272064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.594844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.594846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.594849:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.594855:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829170.594876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.594888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78aa300 x1874663333272192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.594899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333272192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.594908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.594911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.594914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829170.594919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829170.594940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.594951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78a9180 x1874663333272320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.594963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333272320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.594971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.594974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.594978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829170.594984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829170.595003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.595011:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78a8700 x1874663333272448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.595019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333272448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.595024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.595026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.595029:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.595042:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829170.595053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.595060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78a9500 x1874663333272576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.595072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333272576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.595080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.595083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.595087:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.595093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829170.595112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829170.595124:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff9057c78ab100 x1874663333272704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.595132:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333272704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.595137:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.595139:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829170.595141:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333272704:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829170.978505:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829172.002410:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829173.026400:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829174.050417:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829175.074473:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829175.714358:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714367:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829175.714370:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714378:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714388:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714394:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829175.714438:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333272832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714461:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714464:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829175.714465:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714468:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714471:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714476:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829175.714495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333272960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714510:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829175.714512:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714517:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714522:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714527:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829175.714545:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333273088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714557:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714563:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829175.714564:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333272832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829175.714566:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714576:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714596:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714601:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829175.714617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333273216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714624:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714638:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829175.714640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714643:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714646:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714648:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714651:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829175.714659:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333273344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714665:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714668:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829175.714670:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714673:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714676:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829175.714682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272960:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829175.714682:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714689:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829175.714700:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333273472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714707:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829175.714711:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714714:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714717:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829175.714722:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273088:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829175.714722:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829175.714726:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829175.714736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333273600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829175.714745:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829175.714748:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829175.714750:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829175.714753:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10986 00000100:00080000:3.0:1787829175.714757:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829175.714760:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829175.714763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273216:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787829175.714779:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829175.714791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333273728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829175.714799:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829175.714818:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829175.714834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829175.714847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273728:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829176.098478:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829177.122449:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829178.146417:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829178.786503:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829178.786574:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829178.786592:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786633:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786644:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786653:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786662:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786675:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829178.786686:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829178.786690:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb223800 x1874663333273088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829178.786698:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829178.786725:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333273088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786738:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786744:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786749:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829178.786758:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb220000 x1874663333273216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786771:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333273216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786784:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786787:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829178.786793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb220a80 x1874663333273344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333273344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786814:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786817:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829178.786823:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb223b80 x1874663333273472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786840:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333273472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786849:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786855:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829178.786861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb221180 x1874663333272832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786873:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333272832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786881:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786884:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333272832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829178.786894:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb222680 x1874663333272960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333272960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333272960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829178.786958:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb220e00 x1874663333273600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.786972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333273600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.786981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.786984:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.786988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829178.786995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829178] req@ffff9057fb221500 x1874663333273728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.787007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333273728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.787015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.787018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829178.787021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333273728:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829179.170421:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829180.194388:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829180.834326:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834335:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829180.834339:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834347:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834357:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834364:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829180.834394:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333273856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834417:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834420:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829180.834423:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834426:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834430:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834440:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829180.834450:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333273984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834457:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834459:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829180.834460:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834471:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829180.834472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333273856:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829180.834475:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829180.834485:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333274112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834491:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829180.834494:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834497:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834498:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834500:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829180.834511:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333274240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834517:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829180.834521:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834523:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834524:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834527:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829180.834536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333274368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829180.834563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834565:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834568:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834570:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834572:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829180.834579:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333274496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834584:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834587:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829180.834588:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834592:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834593:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829180.834600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333274624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829180.834606:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829180.834608:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829180.834609:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829180.834611:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10991 00000100:00080000:3.0:1787829180.834614:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829180.834619:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829180.834620:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829180.834623:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829180.834630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333274752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829180.834672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb222a00 x1874663333273856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.834692:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333273856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.834701:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.834706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.834710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333273856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829180.834717:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.834741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.834753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb220700 x1874663333273984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.834763:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333273984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.834770:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.834773:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.834776:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333273984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.834800:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.834822:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.834834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb221880 x1874663333274112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.834847:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333274112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.834854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.834858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.834861:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.834867:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829180.834887:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.834898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb223100 x1874663333274240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.834909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333274240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.834917:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.834921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.834924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829180.834930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829180.834952:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.834963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb222d80 x1874663333274368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.834981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333274368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.834989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.834992:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.834996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333274368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829180.835001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.835021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.835032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb221c00 x1874663333274496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.835043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333274496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.835050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.835052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.835055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.835060:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.835077:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.835087:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb222300 x1874663333274624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.835098:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333274624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.835112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.835116:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.835119:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.835125:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829180.835145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829180.835157:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff9057fb220380 x1874663333274752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.835169:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333274752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.835177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.835180:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829180.835184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333274752:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829181.218400:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829182.242446:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829183.266633:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829184.290465:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829185.314432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829185.954320:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954328:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829185.954331:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954339:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954348:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954354:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829185.954387:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333274880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954421:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954424:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829185.954426:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954429:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954431:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954433:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954442:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829185.954451:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333275008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954459:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954462:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829185.954464:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954466:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954468:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954472:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954486:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829185.954494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333274880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829185.954495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333275136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954533:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829185.954535:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954546:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829185.954564:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333275264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954576:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954580:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829185.954582:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954588:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954593:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954598:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829185.954605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829185.954626:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333275392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954638:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954641:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829185.954643:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954651:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00100000:1.0:1787829185.954656:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb220380 x1874663333274880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829185.954659:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954670:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954683:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829185.954699:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333275520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829185.954707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333274880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829185.954709:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954722:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829185.954725:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954741:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954747:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829185.954751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829185.954752:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829185.954768:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787829185.954768:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829185.954784:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333274880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829185.954785:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333275648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829185.954812:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829185.954814:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829185.954817:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829185.954820:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 10996 00000100:00080000:3.0:1787829185.954822:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829185.954824:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829185.954826:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829185.954833:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333275776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829185.954845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.954873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.954884:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb222300 x1874663333275008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.954894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333275008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.954901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.954910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.954913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.954918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.954931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.954939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb221c00 x1874663333275136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.954947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333275136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.954952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.954955:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.954962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.954966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829185.954986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.954998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb222d80 x1874663333275264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.955012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333275264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.955021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.955025:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.955027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829185.955032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829185.955049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.955057:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb223100 x1874663333275392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.955065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333275392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.955071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.955073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.955075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333275392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829185.955079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.955091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.955097:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb221880 x1874663333275520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.955104:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333275520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.955114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.955117:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.955119:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.955123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.955134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.955139:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb220700 x1874663333275648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.955146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333275648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.955150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.955153:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.955155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.955172:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829185.955183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829185.955189:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff9057fb222a00 x1874663333275776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.955197:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333275776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.955202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.955204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829185.955207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333275776:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829186.338463:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829187.362464:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829188.386421:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829189.410460:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829190.434413:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829191.074367:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074379:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829191.074385:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074408:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074423:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.074433:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829191.074520:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333275904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.074577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074580:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.074582:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074586:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074590:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.074593:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074603:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.074616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333276032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829191.074633:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333275904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829191.074640:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074645:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.074647:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074650:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074653:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.074656:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074661:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.074673:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333276160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.074682:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074685:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.074687:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074691:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074694:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.074697:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829191.074713:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333276288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829191.074722:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829191.074732:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.074738:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074744:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074748:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.074751:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074755:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829191.074767:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333276416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829191.074774:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb221500 x1874663333275904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829191.074800:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.074817:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074829:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074834:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829191.074843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333275904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829191.074844:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074873:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829191.074893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829191.074895:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333276544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829191.074903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.074916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333275904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829191.074956:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.074960:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.074962:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.074968:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00080000:3.0:1787829191.074972:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829191.074978:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276032:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829191.074980:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829191.074999:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.075038:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333276672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829191.075056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829191.075099:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.075105:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.075109:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.075117:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11001 00000100:00100000:1.0:1787829191.075120:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb220e00 x1874663333276032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829191.075128:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.075160:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075170:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333276032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829191.075178:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829191.075180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075193:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829191.075201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829191.075209:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333276800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829191.075229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075282:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb222680 x1874663333276160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075294:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333276160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075313:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829191.075322:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829191.075340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075348:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb221180 x1874663333276288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075358:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333276288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075378:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075380:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829191.075386:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829191.075429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075493:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb223b80 x1874663333276416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075546:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333276416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075564:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075568:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075573:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333276416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829191.075582:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829191.075628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075641:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb220a80 x1874663333276544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075653:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333276544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075663:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075666:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829191.075687:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829191.075705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075714:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb220000 x1874663333276672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075743:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333276672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829191.075766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829191.075780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829191.075788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9057fb223800 x1874663333276800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829191.075815:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333276800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829191.075825:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829191.075828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829191.075832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333276800:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829191.458439:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829192.482431:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829193.506432:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829194.530484:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829195.554561:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829196.194300:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194308:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829196.194312:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194321:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194331:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194338:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829196.194376:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333276928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194399:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194402:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.194403:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194406:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194408:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194410:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194418:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829196.194427:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333277056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194435:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194438:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829196.194454:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333276928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829196.194456:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194469:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829196.194484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333277184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194492:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194494:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.194496:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194498:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194501:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194504:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194507:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.194514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333277312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194520:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194522:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.194524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829196.194529:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829196.194530:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194536:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194539:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.194550:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333277440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194558:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.194559:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194563:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194565:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194568:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829196.194578:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb223800 x1874663333276928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829196.194581:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333277568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.194604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194607:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.194608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00100000:1.0:1787829196.194614:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333276928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829196.194616:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194622:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194628:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829196.194633:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829196.194637:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333277696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829196.194638:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194646:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333276928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829196.194652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.194656:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.194657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.194660:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11006 00000100:00080000:3.0:1787829196.194662:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.194673:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829196.194675:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.194681:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333277824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829196.194684:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829196.194713:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194724:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb220000 x1874663333277056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194734:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333277056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.194743:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.194747:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194750:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829196.194757:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829196.194779:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb220a80 x1874663333277184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333277184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.194808:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.194810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829196.194816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829196.194832:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194842:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb223b80 x1874663333277312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333277312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.194875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.194879:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194882:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829196.194889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829196.194907:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb221180 x1874663333277440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333277440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.194928:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.194930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194933:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333277440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829196.194936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829196.194948:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb222680 x1874663333277568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333277568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.194970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.194972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.194974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829196.194977:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829196.194987:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.194992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb220e00 x1874663333277696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.194999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333277696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.195004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.195006:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.195008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829196.195021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829196.195032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829196.195038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9057fb221500 x1874663333277824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829196.195045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333277824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829196.195049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829196.195051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829196.195054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333277824:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829196.578515:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829197.602555:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829198.626394:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829199.650489:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829200.674438:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829201.314341:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314348:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829201.314352:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314375:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829201.314431:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333277952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829201.314453:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314457:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829201.314458:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314462:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314467:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314475:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829201.314486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333278080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829201.314500:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314505:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829201.314508:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829201.314508:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333277952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829201.314517:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314520:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829201.314547:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333278208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829201.314554:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314557:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829201.314558:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314561:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314575:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314578:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829201.314603:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333278336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829201.314610:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829201.314612:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314616:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829201.314617:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314621:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314625:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314628:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314631:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829201.314657:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333278464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829201.314664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314667:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829201.314668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314671:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314673:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314676:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314680:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829201.314682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158dc00 x1874663333277952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829201.314691:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333278592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829201.314720:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333277952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829201.314724:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314731:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829201.314734:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314741:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:1.0:1787829201.314742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829201.314744:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829201.314748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829201.314750:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314756:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829201.314756:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333277952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829201.314787:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333278720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829201.314802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278080:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829201.314803:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829201.314811:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829201.314812:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829201.314817:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11012 00000100:00080000:0.0:1787829201.314820:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829201.314824:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829201.314828:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829201.314838:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333278848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829201.314841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.314864:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158e300 x1874663333278080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.314893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333278080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.314900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.314903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.314914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829201.314919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829201.314934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.314941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158f100 x1874663333278208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.314950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333278208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.314959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.314962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.314966:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829201.314984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829201.315007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.315020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158ca80 x1874663333278336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.315042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333278336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.315053:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.315057:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.315060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829201.315065:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829201.315083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.315092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158df80 x1874663333278464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.315105:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333278464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.315111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.315113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.315116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333278464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829201.315119:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829201.315131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.315173:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158e680 x1874663333278592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.315186:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333278592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.315193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.315196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.315199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829201.315263:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829201.315306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.315319:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158f800 x1874663333278720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.315333:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333278720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.315349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.315352:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.315356:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829201.315364:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829201.315387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829201.315400:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829201/real 1787829201] req@ffff9057f158c000 x1874663333278848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829201.315428:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333278848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829201.315435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829201.315438:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829201.315440:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333278848:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829201.698446:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829202.722481:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829203.746491:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829204.770484:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829205.794492:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829206.434389:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434401:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829206.434404:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434438:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434450:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434457:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829206.434502:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333278976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829206.434528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434531:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829206.434533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434536:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434539:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434561:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434573:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829206.434587:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333279104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829206.434600:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434604:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829206.434624:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333278976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829206.434625:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434640:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434645:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434649:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434655:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829206.434671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333279232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829206.434698:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434701:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829206.434703:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434707:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434710:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434713:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434717:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829206.434724:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333279360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829206.434730:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434733:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829206.434734:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434737:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434739:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434741:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434744:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829206.434749:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333279488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829206.434756:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434759:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787829206.434774:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829206.434780:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434790:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434797:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434802:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434809:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829206.434817:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb222a00 x1874663333278976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829206.434844:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333279616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829206.434846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333278976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829206.434880:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434884:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829206.434889:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434900:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434908:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434914:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829206.434916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829206.434920:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829206.434926:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829206.434933:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333279744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829206.434936:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333278976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829206.434962:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829206.434965:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829206.434967:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829206.434971:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11017 00000100:00080000:3.0:1787829206.434975:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829206.434978:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829206.434982:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829206.434987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829206.434994:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333279872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829206.435019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb220700 x1874663333279104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333279104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829206.435060:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829206.435072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435078:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb221880 x1874663333279232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435088:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333279232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435101:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829206.435111:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829206.435132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb223100 x1874663333279360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435151:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333279360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435159:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435161:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829206.435165:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829206.435178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb222d80 x1874663333279488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435191:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333279488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435200:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333279488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829206.435204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829206.435215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435253:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb221c00 x1874663333279616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435262:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333279616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435274:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435276:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829206.435280:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829206.435293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435300:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb222300 x1874663333279744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435307:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333279744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435313:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435315:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829206.435331:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829206.435342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829206.435348:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829206/real 1787829206] req@ffff9057fb220380 x1874663333279872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829206.435355:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333279872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829206.435360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829206.435362:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829206.435365:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333279872:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829206.818477:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829207.842479:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829208.866390:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829209.890456:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829210.914426:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829211.554346:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554355:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829211.554359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554367:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554375:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829211.554415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333280000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829211.554443:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554445:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829211.554447:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554449:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554452:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554455:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554462:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829211.554470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333280128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829211.554478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554480:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829211.554481:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554484:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554486:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554489:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554491:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829211.554499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333280000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829211.554501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333280256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829211.554512:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554514:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829211.554515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554517:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554520:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554522:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829211.554539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333280384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829211.554550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829211.554554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554556:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554558:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554560:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554564:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829211.554569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829211.554574:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333280512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829211.554582:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554584:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829211.554586:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554589:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554591:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829211.554594:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554597:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829211.554605:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333280640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829211.554610:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb220380 x1874663333280000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829211.554616:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554623:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829211.554626:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554631:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554634:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829211.554640:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333280000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829211.554640:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554643:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829211.554652:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333280768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829211.554655:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829211.554664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829211.554669:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829211.554671:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333280000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829211.554672:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829211.554679:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11022 00000100:00080000:3.0:1787829211.554682:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829211.554685:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280128:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829211.554688:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829211.554694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829211.554703:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333280896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829211.554715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554727:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb222300 x1874663333280128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554736:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333280128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554745:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554747:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829211.554764:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829211.554777:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb221c00 x1874663333280256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333280256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554798:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829211.554804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829211.554815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb222d80 x1874663333280384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333280384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829211.554839:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829211.554849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb223100 x1874663333280512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333280512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333280512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829211.554878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829211.554887:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb221880 x1874663333280640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333280640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829211.554911:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829211.554921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb220700 x1874663333280768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333280768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554941:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554943:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829211.554948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829211.554957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829211.554963:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829211/real 1787829211] req@ffff9057fb222a00 x1874663333280896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829211.554969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333280896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829211.554974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829211.554976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829211.554978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333280896:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829211.938506:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829212.964886:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829213.986488:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829215.010432:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829216.036673:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829216.674369:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674438:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829216.674456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674468:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674481:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674503:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829216.674608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333281024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829216.674658:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674660:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829216.674662:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674665:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674669:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674680:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829216.674689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333281152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829216.674696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674698:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829216.674699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674702:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674721:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674727:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674733:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829216.674748:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333281280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829216.674763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333281024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829216.674787:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674790:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829216.674792:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674797:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674800:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674804:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674810:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829216.674822:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333281408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829216.674848:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674851:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829216.674853:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674857:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674861:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674867:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674882:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829216.674896:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333281536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829216.674906:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829216.674906:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674916:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829216.674920:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674925:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674931:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674936:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674940:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829216.674952:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333281664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829216.674959:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829216.674961:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829216.674962:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.674965:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00080000:2.0:1787829216.674968:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.674973:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.674977:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829216.674979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de0000 x1874663333281024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829216.674989:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333281792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829216.675027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333281024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829216.675044:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829216.675046:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829216.675063:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829216.675076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333281024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829216.675091:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829216.675099:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11027 00000100:00100000:1.0:1787829216.675100:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281152:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829216.675108:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829216.675113:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829216.675118:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829216.675131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829216.675132:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333281920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829216.675160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de1c00 x1874663333281152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675177:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333281152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675194:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829216.675263:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829216.675278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675295:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de1500 x1874663333281280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675323:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333281280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675334:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829216.675339:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829216.675381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675389:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de0700 x1874663333281408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675404:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333281408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675423:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675425:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829216.675429:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829216.675442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675448:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de0a80 x1874663333281536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675456:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333281536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675462:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675464:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675465:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333281536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829216.675470:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829216.675482:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de3800 x1874663333281664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675522:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333281664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675539:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675542:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675546:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829216.675552:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829216.675571:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de1880 x1874663333281792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675605:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333281792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675609:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675611:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675613:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829216.675617:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829216.675640:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829216.675646:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829216] req@ffff9057d0de1180 x1874663333281920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829216.675652:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333281920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829216.675657:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829216.675659:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829216.675662:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333281920:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829217.058487:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829218.082390:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829219.106475:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829220.130394:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829221.154395:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829221.794345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794354:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829221.794358:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829221.794367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794384:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829221.794426:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333282048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829221.794448:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829221.794453:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829221.794460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794467:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829221.794493:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333282176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829221.794503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333282048:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829221.794509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829221.794516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829221.794521:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794525:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794530:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829221.794551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333282304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829221.794559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794563:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829221.794565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829221.794567:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829221.794572:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794577:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794581:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829221.794595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333282432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829221.794603:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829221.794607:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829221.794611:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb221500 x1874663333282048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829221.794620:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794625:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794628:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829221.794633:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333282048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829221.794645:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333282560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829221.794646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.794655:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.794661:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333282048:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829221.794668:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829221.794673:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829221.794679:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794681:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794684:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794687:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829221.794695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333282688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829221.794701:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282176:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829221.794706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794710:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829221.794712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829221.794716:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794718:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794721:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794725:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829221.794735:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829221.794736:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333282816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829221.794746:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829221.794749:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829221.794752:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829221.794753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb220e00 x1874663333282176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829221.794757:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11032 00000100:00080000:3.0:1787829221.794762:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829221.794769:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829221.794775:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829221.794780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333282176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829221.794791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333282944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829221.794792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.794801:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.794809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829221.794821:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829221.794852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.794861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb222680 x1874663333282304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.794873:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333282304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.794881:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.794884:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.794888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829221.794895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829221.794916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.794927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb221180 x1874663333282432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.794940:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333282432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.794949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.794953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.794956:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829221.794963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829221.794979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.794987:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb223b80 x1874663333282560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.794999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333282560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.795007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.795011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.795014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333282560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829221.795032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829221.795063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.795072:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb220a80 x1874663333282688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.795080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333282688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.795086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.795088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.795090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829221.795094:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829221.795107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.795117:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb220000 x1874663333282816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.795129:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333282816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.795144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.795148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.795151:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829221.795158:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829221.795173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829221.795181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff9057fb223800 x1874663333282944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829221.795188:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333282944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829221.795194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829221.795196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829221.795198:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333282944:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829222.178410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829223.202489:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829224.226395:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829225.250410:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829226.274534:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829226.914336:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914344:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829226.914348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914357:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914367:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829226.914412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333283072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829226.914443:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914446:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914448:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914451:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914459:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829226.914469:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333283200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914477:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914480:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829226.914481:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914487:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00100000:1.0:1787829226.914489:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333283072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829226.914490:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914497:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829226.914510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333283328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914517:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914519:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829226.914521:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914523:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914525:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914527:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914530:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829226.914539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333283456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829226.914550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914553:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914556:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914558:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829226.914570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333283584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914579:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829226.914580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829226.914584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829226.914587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914591:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914593:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914595:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914597:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829226.914603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333283712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914611:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829226.914612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914614:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914618:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829226.914626:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333283840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829226.914631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829226.914633:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829226.914634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829226.914636:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11037 00000100:00080000:2.0:1787829226.914639:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829226.914642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829226.914645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829226.914652:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333283968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829226.914682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de1f80 x1874663333283072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.914704:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333283072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.914717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.914721:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.914725:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333283072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829226.914752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829226.914780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.914794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de3100 x1874663333283200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.914807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333283200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.914817:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.914820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.914824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829226.914830:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829226.914852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.914865:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de2a00 x1874663333283328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.914878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333283328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.914886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.914890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.914894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829226.914900:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829226.914923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.914935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de0e00 x1874663333283456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.914948:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333283456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.914956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.914959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.914961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829226.914966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829226.914983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.914992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de1180 x1874663333283584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.915003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333283584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.915010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.915023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.915027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333283584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829226.915032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829226.915050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.915059:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de1880 x1874663333283712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.915070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333283712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.915084:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.915087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.915090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829226.915096:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829226.915114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.915125:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de3800 x1874663333283840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.915138:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333283840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.915146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.915149:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.915152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829226.915157:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829226.915177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829226.915190:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff9057d0de0a80 x1874663333283968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829226.915202:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333283968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829226.915210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829226.915213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829226.915217:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333283968:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829227.298394:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829228.322421:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829229.346361:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829230.370450:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829231.394361:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829232.034333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829232.034348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034358:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034367:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034381:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829232.034430:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333284096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.034471:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034475:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.034477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034482:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034488:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034493:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034505:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.034521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333284224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.034532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034537:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.034541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829232.034544:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333284096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829232.034548:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034556:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034562:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.034576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333284352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.034583:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034586:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.034587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034590:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034594:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034596:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.034608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333284480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.034615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.034619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034622:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034626:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034628:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034633:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.034644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333284608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829232.034653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829232.034664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034670:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.034672:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034680:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034685:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034690:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034697:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.034725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333284736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829232.034726:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb223800 x1874663333284096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829232.034788:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034791:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.034796:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034801:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00080000:3.0:1787829232.034807:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034810:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829232.034817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333284096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829232.034820:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.034840:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333284864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829232.034844:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829232.034850:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.034855:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829232.034857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829232.034858:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.034875:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11042 00000100:00100000:1.0:1787829232.034876:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333284096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829232.034880:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.034890:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829232.034896:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.034911:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333284992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829232.034921:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829232.034958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.034977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb220000 x1874663333284224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.034990:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333284224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829232.035016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829232.035038:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb220a80 x1874663333284352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035062:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333284352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035075:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035078:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829232.035086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829232.035108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035120:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb223b80 x1874663333284480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333284480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035142:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035146:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035149:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829232.035156:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829232.035178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb221180 x1874663333284608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035247:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333284608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035256:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035259:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333284608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829232.035263:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829232.035281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035289:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb222680 x1874663333284736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035297:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333284736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035305:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035307:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829232.035310:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829232.035321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035327:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb220e00 x1874663333284864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035334:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333284864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035346:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035348:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829232.035365:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829232.035376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829232.035382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829232] req@ffff9057fb221500 x1874663333284992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829232.035389:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333284992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829232.035395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829232.035397:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829232.035399:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333284992:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829232.418441:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829233.442480:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829234.466429:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829235.490341:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829236.514392:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829237.154295:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154305:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829237.154308:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154318:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154330:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154337:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829237.154374:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333285120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.154398:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154401:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.154402:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154406:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154408:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154410:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154419:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829237.154427:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333285248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.154434:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154437:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.154438:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154441:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154443:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154449:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829237.154458:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333285376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.154467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829237.154483:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333285120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829237.154507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.154510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154515:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154518:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154522:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829237.154540:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333285504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.154548:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154551:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.154552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154556:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154561:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829237.154566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829237.154570:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829237.154586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333285632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.154592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.154603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154610:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00100000:1.0:1787829237.154611:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb222a00 x1874663333285120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.154617:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829237.154646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333285120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.154655:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333285760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829237.154664:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.154664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.154679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154684:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:1.0:1787829237.154685:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829237.154687:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154695:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829237.154701:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333285120:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829237.154701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829237.154713:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333285888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829237.154721:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285248:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829237.154725:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.154736:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.154738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.154741:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11047 00000100:00080000:3.0:1787829237.154744:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.154747:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829237.154751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829237.154766:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829237.154766:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333286016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829237.154789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb220700 x1874663333285248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.154803:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333285248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.154809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.154812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.154815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829237.154841:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829237.154862:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.154874:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb221880 x1874663333285376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.154887:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333285376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.154894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.154896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.154899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829237.154903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829237.154919:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.154927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb223100 x1874663333285504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.154935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333285504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.154940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.154942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.154943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829237.154947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829237.154957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.154962:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb222d80 x1874663333285632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.154969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333285632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.154974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.154976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.154978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333285632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829237.154981:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333285760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829237.154992:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.154997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb221c00 x1874663333285760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.155004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333285760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.155013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.155015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.155017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333285760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829237.155021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333285888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829237.155031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.155037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb222300 x1874663333285888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.155043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333285888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.155047:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.155050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.155052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333285888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829237.155055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829237.155065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829237.155071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9057fb220380 x1874663333286016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829237.155077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333286016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829237.155082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829237.155084:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829237.155087:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286016:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829237.538371:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829238.562373:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829239.586360:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829240.610441:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829241.634365:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829242.274275:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274285:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829242.274289:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274299:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274309:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274317:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829242.274358:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333286144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274388:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274392:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829242.274395:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274400:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274404:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274409:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274420:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829242.274435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333286272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829242.274448:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274451:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274454:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274457:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274461:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829242.274475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333286144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829242.274475:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333286400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274509:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829242.274510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274513:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274515:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274518:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829242.274532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333286528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274539:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274542:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829242.274543:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274546:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274550:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274552:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274555:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829242.274570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333286656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274577:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829242.274582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274585:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274589:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274593:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829242.274605:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333286784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829242.274612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829242.274619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829242.274626:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829242.274630:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274640:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274662:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274671:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829242.274690:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333286912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829242.274702:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286400:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829242.274708:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829242.274713:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829242.274715:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829242.274720:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11053 00000100:00080000:3.0:1787829242.274725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829242.274730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829242.274736:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829242.274739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829242.274756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333287040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829242.274772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829242.274806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829242.274843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829242.274867:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287040:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829242.658427:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829243.682397:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829244.706407:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829245.346574:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829245.346646:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829245.346678:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346741:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346751:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346758:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346769:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346778:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346802:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829245.346814:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787829245.346818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb221c00 x1874663333286400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.346850:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333286400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.346862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.346867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.346872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829245.346880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb222d80 x1874663333286528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.346897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333286528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.346906:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.346910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.346914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829245.346922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb223100 x1874663333286656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.346936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333286656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.346945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.346949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.346952:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829245.346960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb221880 x1874663333286784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.346974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333286784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.346983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.346986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.346990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829245.346998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb220380 x1874663333286144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.347014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333286144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.347023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.347028:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.347038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333286144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829245.347046:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb222300 x1874663333286272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.347061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333286272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.347069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.347073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.347076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333286272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829245.347136:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb220700 x1874663333286912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.347152:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333286912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.347163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.347168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.347171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333286912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829245.347179:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829242/real 1787829245] req@ffff9057fb222a00 x1874663333287040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829245.347193:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333287040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829245.347202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829245.347205:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829245.347209:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287040:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787829245.730458:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829246.754454:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829247.394354:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394365:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829247.394370:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394392:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394405:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394413:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829247.394452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333287168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829247.394485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829247.394506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394518:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.394530:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829247.394547:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333287296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829247.394561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829247.394568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394572:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394577:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394586:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829247.394601:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333287168:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787829247.394610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829247.394635:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333287424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829247.394652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394672:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829247.394674:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394679:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394684:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394689:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.394696:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829247.394711:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829247.394714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333287552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829247.394730:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394735:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829247.394738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394745:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394762:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394767:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.394774:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829247.394791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333287680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829247.394806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de0700 x1874663333287168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829247.394806:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.394823:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829247.394832:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.394847:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394855:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829247.394862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333287168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829247.394862:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.394874:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829247.394899:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333287808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829247.394900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.394916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829247.394929:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829247.394932:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333287168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829247.394938:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829247.394947:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829247.394960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287296:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829247.394960:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.394970:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.394978:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.394984:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829247.395002:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333287936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829247.395011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829247.395041:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829247.395046:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829247.395052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de1500 x1874663333287296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829247.395053:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829247.395069:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11058 00000100:00080000:2.0:1787829247.395077:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829247.395105:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829247.395112:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829247.395121:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333287296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829247.395136:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333288064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829247.395195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395211:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829247.395302:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829247.395335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395357:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de1c00 x1874663333287424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395372:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333287424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395387:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395391:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829247.395398:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829247.395419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395431:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de0000 x1874663333287552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395443:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333287552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395462:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395466:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829247.395474:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829247.395497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395511:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de0a80 x1874663333287680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395522:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333287680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395534:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395538:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333287680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829247.395544:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829247.395566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395579:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de3800 x1874663333287808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395592:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333287808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395601:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395604:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395608:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829247.395615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829247.395637:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395652:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de1880 x1874663333287936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395676:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333287936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395686:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395690:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395693:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333287936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829247.395701:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829247.395723:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829247.395736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829247/real 1787829247] req@ffff9057d0de1180 x1874663333288064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829247.395747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333288064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829247.395756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829247.395760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829247.395763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288064:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829247.778665:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829248.802423:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829249.826333:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829250.850421:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829251.874408:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829252.514365:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514379:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829252.514383:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514395:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514407:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514417:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829252.514471:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333288192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829252.514502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829252.514509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514519:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514524:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514536:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829252.514551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333288320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829252.514559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333288192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829252.514566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829252.514574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514579:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514584:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514588:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829252.514613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333288448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829252.514621:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514624:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829252.514626:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829252.514631:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829252.514634:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829252.514657:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333288576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829252.514664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514667:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829252.514669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514671:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514674:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829252.514677:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb221500 x1874663333288192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829252.514679:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829252.514706:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333288704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829252.514713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829252.514717:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333288192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829252.514717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829252.514721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514729:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514733:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829252.514734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.514740:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829252.514743:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829252.514744:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333288192:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829252.514751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829252.514761:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333288832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829252.514767:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829252.514770:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829252.514772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829252.514775:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288320:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829252.514777:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514783:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514787:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514790:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829252.514799:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333288960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829252.514804:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829252.514815:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829252.514820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb220e00 x1874663333288320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829252.514820:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829252.514824:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829252.514839:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11063 00000100:00080000:3.0:1787829252.514844:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829252.514849:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829252.514852:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829252.514854:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333288320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.514862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829252.514864:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333289088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829252.514869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.514877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829252.514885:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829252.514900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.514911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb222680 x1874663333288448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.514924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333288448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.514939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.514943:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.514958:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829252.514965:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829252.514982:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.514990:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb221180 x1874663333288576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.514998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333288576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.515004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.515006:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.515008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829252.515011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829252.515021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.515027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb223b80 x1874663333288704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.515034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333288704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.515039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.515041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.515043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333288704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829252.515063:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829252.515089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.515097:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb220a80 x1874663333288832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.515104:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333288832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.515110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.515112:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.515114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829252.515128:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829252.515138:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.515156:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb220000 x1874663333288960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.515164:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333288960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.515169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.515171:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.515173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333288960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829252.515177:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829252.515187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829252.515194:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829252/real 1787829252] req@ffff9057fb223800 x1874663333289088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829252.515202:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333289088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829252.515207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829252.515208:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829252.515210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289088:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829252.898461:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829253.922496:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829254.946730:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829255.970412:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829256.994419:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829257.634350:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634359:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829257.634370:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634380:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634391:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634399:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829257.634437:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333289216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829257.634468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829257.634495:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333289344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829257.634507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634509:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634511:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634518:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829257.634524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333289472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829257.634538:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333289216:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829257.634539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829257.634542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634554:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634560:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634581:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829257.634603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333289600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634611:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634615:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829257.634616:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634622:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829257.634639:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333289728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634650:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829257.634651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634670:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634673:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829257.634677:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829257.634682:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634687:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829257.634699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333289856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634710:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829257.634712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634719:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634725:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829257.634729:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829257.634735:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de0e00 x1874663333289216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829257.634761:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333289984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829257.634780:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829257.634783:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829257.634785:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829257.634788:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11068 00000100:00080000:2.0:1787829257.634791:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829257.634797:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333289216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829257.634799:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829257.634806:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829257.634822:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333290112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829257.634831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.634849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.634856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333289216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829257.634866:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.634903:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.634916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de2a00 x1874663333289344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.634929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333289344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.634945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.634961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.634965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.634994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.635031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de3100 x1874663333289472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333289472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.635080:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829257.635099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635108:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de1f80 x1874663333289600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333289600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635123:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635126:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829257.635130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829257.635142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635148:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de1180 x1874663333289728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635155:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333289728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635162:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635164:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333289728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829257.635168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.635178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635184:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de1880 x1874663333289856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635190:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333289856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635197:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.635203:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.635213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635242:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de3800 x1874663333289984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635251:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333289984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635259:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635261:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333289984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.635265:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333290112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829257.635278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829257.635285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff9057d0de0a80 x1874663333290112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.635293:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333290112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.635298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.635300:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829257.635302:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333290112:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829258.018386:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829259.042466:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829260.066385:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829261.090470:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829262.114418:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829262.754358:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754368:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829262.754372:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754381:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754392:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754410:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829262.754447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333290240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.754469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829262.754473:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754477:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754479:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754481:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829262.754500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333290368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.754507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.754511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754515:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829262.754523:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333290240:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787829262.754524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829262.754536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333290496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.754543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829262.754547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754550:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754553:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754556:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829262.754570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333290624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.754584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754587:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.754589:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754592:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754594:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754597:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829262.754605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829262.754611:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333290752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.754619:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829262.754624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754626:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754629:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754631:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829262.754641:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333290880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829262.754653:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158c000 x1874663333290240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829262.754657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.754669:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754676:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11073 00000100:00080000:0.0:1787829262.754692:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.754700:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754706:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829262.754707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333290240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.754719:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829262.754724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333291008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829262.754725:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.754731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333290240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829262.754742:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290368:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829262.754751:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.754754:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.754756:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.754761:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11073 00000800:00000100:1.0:1787829262.754764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829262.754775:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829262.754776:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158f800 x1874663333290368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829262.754785:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829262.754793:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829262.754793:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333290368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.754813:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.754818:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829262.754819:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333291136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829262.754823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.754871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829262.754894:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.754902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158e680 x1874663333290496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.754910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333290496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.754920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.754922:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.754924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.754927:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829262.754939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.754945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158df80 x1874663333290624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.754952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333290624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.754961:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.754963:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.754966:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829262.754969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829262.754996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.755004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158ca80 x1874663333290752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.755012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333290752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.755044:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.755046:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.755049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333290752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829262.755053:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333290880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829262.755068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.755075:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158f100 x1874663333290880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.755082:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333290880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.755088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.755090:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.755092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333290880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.755095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829262.755106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.755112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158e300 x1874663333291008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.755120:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333291008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.755125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.755127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.755129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.755133:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829262.755143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829262.755150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff9057f158dc00 x1874663333291136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.755158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333291136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.755163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.755165:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829262.755167:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291136:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829263.138464:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829264.162376:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829265.186417:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829266.210367:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829267.234582:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829267.874317:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874328:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829267.874332:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874343:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874355:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874362:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829267.874408:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333291264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874455:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874458:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829267.874460:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874464:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874467:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829267.874490:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333291392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874498:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874501:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829267.874502:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874505:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874507:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874510:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874513:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829267.874520:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333291264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829267.874522:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333291520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874540:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829267.874544:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874546:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874549:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874551:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874554:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829267.874561:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333291648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829267.874571:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874573:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874588:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874591:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829267.874602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333291776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829267.874608:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829267.874614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829267.874619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874622:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874629:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829267.874640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333291904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829267.874650:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874652:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874654:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829267.874656:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829267.874668:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158d880 x1874663333291264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829267.874669:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333292032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829267.874704:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829267.874707:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829267.874709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829267.874713:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11078 00000100:00080000:0.0:1787829267.874716:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829267.874728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333291264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829267.874731:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829267.874735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829267.874743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333292160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829267.874748:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.874758:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.874765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333291264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829267.874771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.874790:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.874801:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158ce00 x1874663333291392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.874811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333291392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.874821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.874835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.874837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.874858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.874869:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.874875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158c700 x1874663333291520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.874883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333291520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.874888:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.874890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.874892:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.874895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829267.874906:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.874913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158ea00 x1874663333291648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.874920:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333291648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.874925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.874927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.874929:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829267.874933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829267.874961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.874972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158d180 x1874663333291776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.874984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333291776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.874990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.875003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.875005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333291776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829267.875009:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.875023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.875030:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158d500 x1874663333291904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.875037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333291904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.875042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.875044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.875045:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333291904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.875049:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.875059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.875065:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158ed80 x1874663333292032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.875073:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333292032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.875078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.875079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.875081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.875084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829267.875094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829267.875099:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff9057f158fb80 x1874663333292160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.875106:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333292160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.875111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.875113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829267.875115:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292160:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829268.258388:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829269.282544:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829270.306399:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829271.330451:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829272.354393:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829272.994321:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994331:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829272.994335:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994344:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994354:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994361:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829272.994400:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333292288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994429:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829272.994430:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994436:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994438:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829272.994447:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829272.994458:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333292416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994466:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829272.994470:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994473:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994477:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994484:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829272.994487:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333292288:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787829272.994488:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829272.994508:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333292544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994516:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829272.994520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994523:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994525:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994528:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829272.994531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829272.994539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333292672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994546:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829272.994550:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994554:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994566:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994569:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829272.994572:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829272.994583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829272.994584:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333292800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994602:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829272.994607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.994613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994615:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994617:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829272.994621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829272.994644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158fb80 x1874663333292288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829272.994646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333292928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829272.994718:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829272.994723:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333292288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829272.994727:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994740:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.994751:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829272.994751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.994777:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:0.0:1787829272.994781:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829272.994783:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333292288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829272.994793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829272.994799:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333293056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.994816:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.994826:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829272.994826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829272.994835:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829272.994842:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158ed80 x1874663333292416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829272.994848:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11083 00000100:00080000:0.0:1787829272.994863:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829272.994865:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333292416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829272.994879:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829272.994880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.994890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.994895:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292416:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787829272.994896:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829272.994938:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333293184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829272.994949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829272.994991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158d500 x1874663333292544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333292544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995027:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995030:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829272.995039:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829272.995062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158d180 x1874663333292672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333292672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995097:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829272.995107:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829272.995128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158ea00 x1874663333292800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995145:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333292800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995153:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995156:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333292800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829272.995160:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829272.995183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995190:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158c700 x1874663333292928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995197:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333292928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995205:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333292928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829272.995211:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829272.995258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995266:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158ce00 x1874663333293056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995274:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333293056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995284:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829272.995297:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829272.995325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829272.995338:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff9057f158d880 x1874663333293184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.995352:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333293184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.995361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.995363:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829272.995366:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293184:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829273.378405:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829274.402366:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829275.426446:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829276.450541:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829277.474437:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829278.114351:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114361:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829278.114366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114377:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114389:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114397:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829278.114453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333293312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114488:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114492:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.114495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114501:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114505:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114510:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.114554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333293440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.114574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114578:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114584:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829278.114585:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333293312:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829278.114591:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114608:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.114618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333293568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.114631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114634:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114641:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114645:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829278.114654:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333293696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114662:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.114666:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114669:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114674:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114677:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114681:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829278.114689:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333293824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.114717:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114720:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114723:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114726:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.114737:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333293952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114744:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114747:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.114748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114751:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114753:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114756:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829278.114767:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829278.114771:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333294080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.114789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.114793:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.114795:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.114799:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11088 00000100:00080000:1.0:1787829278.114802:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.114805:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829278.114817:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829278.114828:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333294208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829278.114840:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78a8a80 x1874663333293312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.114868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333293312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.114901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.114905:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.114909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333293312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829278.114916:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829278.114939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.114950:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78ab100 x1874663333293440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.114961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333293440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.114980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.114983:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.114986:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829278.115011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829278.115026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78a9500 x1874663333293568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333293568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115068:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829278.115076:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829278.115090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115097:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78a8700 x1874663333293696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115107:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333293696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115116:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115119:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829278.115139:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829278.115163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78a9180 x1874663333293824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115178:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333293824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115185:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115188:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333293824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829278.115193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829278.115205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115212:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78aa300 x1874663333293952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115220:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333293952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115228:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115230:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333293952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829278.115234:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829278.115276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115284:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78a9c00 x1874663333294080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115292:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333294080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115301:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115303:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829278.115307:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829278.115319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829278.115326:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9057c78aa680 x1874663333294208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829278.115335:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333294208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829278.115341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829278.115343:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829278.115345:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294208:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829278.498417:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829279.522440:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829280.546357:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829281.570415:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829282.594371:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829283.234354:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234366:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829283.234371:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234382:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234393:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234400:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829283.234440:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333294336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829283.234472:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829283.234478:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234483:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234487:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829283.234517:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333294464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829283.234525:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829283.234529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234532:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234535:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829283.234551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333294592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829283.234557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333294336:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829283.234561:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234565:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829283.234566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234569:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234571:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234573:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829283.234586:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333294720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829283.234592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829283.234596:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234598:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234614:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234617:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829283.234624:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829283.234629:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333294848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829283.234637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234651:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829283.234653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234656:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234668:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234674:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829283.234683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333294976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829283.234691:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb223800 x1874663333294336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829283.234692:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829283.234702:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234710:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234713:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829283.234715:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234719:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829283.234720:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333294336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829283.234731:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333295104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829283.234732:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.234741:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829283.234745:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829283.234750:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829283.234752:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333294336:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829283.234753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829283.234760:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11093 00000100:00080000:3.0:1787829283.234763:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829283.234765:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294464:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829283.234769:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829283.234775:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829283.234784:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333295232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829283.234792:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.234803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb220000 x1874663333294464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.234812:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333294464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.234818:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.234821:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.234823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829283.234859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829283.234873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.234890:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb220a80 x1874663333294592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.234897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333294592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.234903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.234905:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.234907:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829283.234910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829283.234921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.234926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb223b80 x1874663333294720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.234933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333294720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.234938:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.234939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.234942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829283.234945:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829283.234955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.234961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb221180 x1874663333294848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.234967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333294848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.234973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.234974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.234976:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333294848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829283.234979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829283.234989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.234994:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb222680 x1874663333294976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.235001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333294976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.235006:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.235008:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.235009:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333294976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829283.235013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333295104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829283.235022:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.235028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb220e00 x1874663333295104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.235035:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333295104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.235040:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.235041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.235043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333295104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829283.235046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333295232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829283.235056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829283.235061:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829283/real 1787829283] req@ffff9057fb221500 x1874663333295232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829283.235067:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333295232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829283.235072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829283.235074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829283.235076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333295232:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829283.618464:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829284.642498:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829285.666380:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829286.690444:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829287.714432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829288.354384:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354393:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829288.354397:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354406:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354417:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829288.354424:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829288.354461:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333295360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354486:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354489:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829288.354490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354494:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354496:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829288.354499:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354508:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829288.354518:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333295488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354526:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354529:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829288.354530:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354533:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354536:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829288.354539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829288.354550:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333295360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829288.354558:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333295616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354566:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354568:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829288.354570:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354575:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354578:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829288.354581:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354584:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829288.354594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333295744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354604:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829288.354605:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354607:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354618:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829288.354621:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829288.354629:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354632:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829288.354647:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333295872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354659:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354662:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829288.354663:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354669:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354674:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829288.354678:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb222a00 x1874663333295360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829288.354679:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354690:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829288.354707:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333296000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829288.354717:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829288.354721:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333295360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829288.354725:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829288.354729:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354744:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354753:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829288.354758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829288.354759:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354771:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829288.354772:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.354785:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333295360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829288.354791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333296128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829288.354814:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295488:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829288.354836:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829288.354839:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829288.354842:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829288.354847:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11099 00000100:00080000:3.0:1787829288.354851:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829288.354856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829288.354857:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829288.354867:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829288.354877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb220700 x1874663333295488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829288.354887:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333296256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829288.354892:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333295488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.354948:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.354953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.354957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829288.354985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829288.354999:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb222300 x1874663333295616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333295616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355028:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829288.355032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829288.355044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb220380 x1874663333295744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333295744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355062:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829288.355070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829288.355080:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355085:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb221880 x1874663333295872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355092:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333295872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333295872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829288.355110:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829288.355131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355143:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb223100 x1874663333296000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355154:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333296000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355165:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355169:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829288.355175:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829288.355195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb222d80 x1874663333296128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355212:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333296128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355255:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355257:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829288.355262:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829288.355276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829288.355284:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829288/real 1787829288] req@ffff9057fb221c00 x1874663333296256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829288.355291:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333296256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829288.355296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829288.355298:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829288.355300:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333296256:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829288.738445:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829289.762409:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829290.786472:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829291.810445:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829292.834398:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829293.474320:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474328:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829293.474332:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474342:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474351:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474358:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829293.474395:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333296384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474419:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474433:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829293.474434:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474438:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474445:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474449:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829293.474457:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829293.474466:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333296512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474476:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829293.474477:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333296384:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829293.474480:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829293.474483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474488:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474499:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474501:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829293.474504:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829293.474514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333296640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474521:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474524:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829293.474525:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474528:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474530:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474533:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829293.474536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829293.474542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333296768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474551:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829293.474553:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474557:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11104 00000800:00000100:2.0:1787829293.474558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829293.474560:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474564:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829293.474567:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829293.474575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333296896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474581:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474584:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829293.474585:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474588:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474591:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474594:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474598:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158dc00 x1874663333296384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829293.474599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829293.474611:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333297024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474619:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829293.474620:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333296384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829293.474623:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829293.474627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474635:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474641:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829293.474643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829293.474644:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829293.474650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829293.474650:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333296384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829293.474661:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333297152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829293.474669:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829293.474671:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829293.474672:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829293.474675:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11104 00000100:00080000:0.0:1787829293.474677:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829293.474680:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296512:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787829293.474685:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829293.474698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333297280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829293.474708:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158e300 x1874663333296512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333296512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474735:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474737:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829293.474743:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829293.474753:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474759:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158f100 x1874663333296640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474782:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333296640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474788:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474790:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829293.474796:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829293.474811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158ca80 x1874663333296768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333296768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474832:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829293.474838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829293.474849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474854:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158df80 x1874663333296896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333296896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474869:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333296896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829293.474872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829293.474882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158e680 x1874663333297024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474903:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333297024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829293.474916:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829293.474927:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.474934:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158f800 x1874663333297152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.474941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333297152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.474947:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.474950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.474953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829293.474973:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829293.474994:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829293.475005:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829293/real 1787829293] req@ffff9057f158c000 x1874663333297280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829293.475029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333297280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829293.475038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829293.475041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829293.475044:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333297280:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829293.858391:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829294.882390:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829295.906456:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829296.930383:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829297.954486:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829298.594427:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594438:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829298.594441:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594451:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594462:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594468:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829298.594521:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333297408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829298.594556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829298.594563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594568:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594572:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594578:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829298.594599:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829298.594630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333297536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829298.594641:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829298.594649:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594664:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594667:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594675:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829298.594677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333297408:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829298.594680:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829298.594701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333297664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829298.594711:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594715:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829298.594716:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594721:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594726:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829298.594749:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829298.594790:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333297792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829298.594796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594799:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829298.594801:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594803:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594807:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594814:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829298.594816:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829298.594833:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829298.594847:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333297920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829298.594856:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594859:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829298.594861:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594865:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594867:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.594869:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829298.594874:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829298.594889:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333298048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829298.594890:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb221c00 x1874663333297408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829298.594960:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.594971:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829298.594976:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.594988:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.594995:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829298.595003:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829298.595009:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829298.595032:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333298176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829298.595038:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333297408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829298.595047:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829298.595059:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829298.595064:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829298.595076:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11109 00000100:00080000:3.0:1787829298.595085:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829298.595090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829298.595091:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829298.595104:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829298.595105:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595118:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333297408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829298.595123:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333298304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829298.595150:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829298.595177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595189:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb222d80 x1874663333297536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595215:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333297536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595225:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595227:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829298.595233:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829298.595247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595254:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb223100 x1874663333297664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595263:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333297664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595277:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595279:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829298.595314:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829298.595330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595339:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb221880 x1874663333297792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595348:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333297792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595357:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595359:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829298.595364:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829298.595376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb220380 x1874663333297920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595391:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333297920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595398:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595401:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333297920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829298.595406:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829298.595418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595424:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb222300 x1874663333298048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595432:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333298048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595440:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595442:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829298.595448:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829298.595467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595480:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb220700 x1874663333298176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595494:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333298176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595519:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595524:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595528:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829298.595536:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829298.595557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829298.595567:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff9057fb222a00 x1874663333298304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829298.595589:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333298304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829298.595600:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829298.595603:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829298.595606:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333298304:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829298.978397:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829300.002437:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829301.026423:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829302.050445:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829303.074375:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829303.714295:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714307:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829303.714312:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714325:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714337:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714356:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829303.714396:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333298432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.714424:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829303.714428:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714437:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714440:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714444:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714451:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829303.714460:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333298560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.714472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714475:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829303.714477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714481:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714491:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829303.714499:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333298432:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829303.714506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829303.714523:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333298688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.714540:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714543:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829303.714544:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714547:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714560:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829303.714565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714566:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829303.714586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333298816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.714594:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714596:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829303.714597:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714600:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714604:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714607:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714611:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829303.714618:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb221500 x1874663333298432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829303.714621:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333298944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.714642:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714645:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829303.714646:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714648:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714650:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714653:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829303.714660:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333298432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829303.714665:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333299072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829303.714680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829303.714680:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714686:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829303.714687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829303.714688:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829303.714705:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333298432:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829303.714708:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714712:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829303.714715:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298560:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829303.714719:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714726:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829303.714736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333299200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829303.714760:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829303.714761:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.714765:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829303.714767:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.714771:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11114 00000100:00080000:3.0:1787829303.714774:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.714777:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829303.714780:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829303.714781:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb220e00 x1874663333298560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829303.714791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333299328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829303.714804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333298560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.714816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.714820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.714826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829303.714852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829303.714871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.714881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb222680 x1874663333298688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.714893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333298688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.714902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.714906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.714909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829303.714915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829303.714931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.714941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb221180 x1874663333298816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.714949:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333298816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.714954:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.714956:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.714959:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829303.714964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829303.714983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.714994:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb223b80 x1874663333298944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.715007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333298944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.715016:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.715020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.715023:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333298944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829303.715030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829303.715047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.715058:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb220a80 x1874663333299072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.715070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333299072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.715078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.715082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.715085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829303.715091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829303.715105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.715114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb220000 x1874663333299200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.715127:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333299200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.715135:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.715138:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.715142:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829303.715147:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829303.715161:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829303.715170:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff9057fb223800 x1874663333299328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829303.715182:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333299328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829303.715190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829303.715193:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829303.715197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333299328:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829304.098446:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829305.122462:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829306.146431:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829307.170361:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829308.194418:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829308.834314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829308.834329:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834344:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834356:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834363:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829308.834400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333299456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834425:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834429:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829308.834430:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834436:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834439:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834446:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829308.834456:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333299584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829308.834468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834473:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834475:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829308.834483:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333299456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829308.834488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333299712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834499:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829308.834500:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834503:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834506:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834508:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829308.834521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333299840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829308.834532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834534:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834539:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829308.834549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333299968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834556:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834559:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829308.834560:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834563:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834565:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834567:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829308.834579:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829308.834582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333300096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834594:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829308.834597:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834604:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829308.834606:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829308.834618:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829308.834621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333300224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.834637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829308.834639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829308.834641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.834644:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11119 00000100:00080000:1.0:1787829308.834648:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829308.834651:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299840:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829308.834654:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829308.834673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829308.834686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333300352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829308.834703:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829308.834730:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829308.834759:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829308.834780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300352:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829309.218399:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829310.242431:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787829311.266380:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829311.906454:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829311.906503:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829311.906525:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906568:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906581:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906591:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906603:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906615:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829311.906617:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78abb80 x1874663333299712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829311.906628:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829311.906660:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829311.906671:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333299712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906686:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829311.906701:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78a8000 x1874663333299840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906722:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333299840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906731:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906735:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906738:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829311.906748:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78aad80 x1874663333299968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906761:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333299968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906769:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906772:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906775:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829311.906782:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78a9880 x1874663333300096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906792:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333300096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906799:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829311.906814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78aaa00 x1874663333299456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333299456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906836:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333299456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829311.906847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78ab480 x1874663333299584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333299584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906871:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333299584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829311.906907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78ab800 x1874663333300224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906919:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333300224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829311.906940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829311] req@ffff9057c78a8380 x1874663333300352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829311.906951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333300352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829311.906958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829311.906961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829311.906965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333300352:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829312.290533:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829313.314439:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829313.954290:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829313.954300:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829313.954303:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954313:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954325:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954331:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829313.954365:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333300480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829313.954390:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829313.954393:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829313.954394:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954398:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954400:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954404:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954413:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829313.954441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333300608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829313.954441:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333300480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829313.954461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829313.954465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829313.954467:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954475:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829313.954500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333300736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829313.954511:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829313.954515:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829313.954519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829313.954519:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829313.954527:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954532:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954536:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829313.954556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333300864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829313.954566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829313.954568:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78a9f80 x1874663333300480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.954570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829313.954574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954585:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954591:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954595:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829313.954609:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333300480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.954615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333300992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829313.954637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.954647:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829313.954648:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829313.954654:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333300480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829313.954655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829313.954660:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954668:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954673:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954678:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954683:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829313.954692:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829313.954700:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333301120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829313.954714:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829313.954718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829313.954720:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829313.954725:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954729:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829313.954733:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829313.954738:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954745:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829313.954755:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78a8380 x1874663333300608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.954767:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333301248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829313.954774:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333300608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.954789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829313.954792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.954796:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829313.954800:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829313.954802:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829313.954808:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300608:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829313.954810:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11124 00000100:00080000:1.0:1787829313.954820:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829313.954825:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829313.954829:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829313.954838:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829313.954854:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333301376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829313.954857:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.954875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78ab800 x1874663333300736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.954891:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333300736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.954900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.954903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.954906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829313.954912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829313.954933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.954944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78ab480 x1874663333300864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.954955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333300864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.954962:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.954965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.954968:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829313.954974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829313.954998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.955009:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78aaa00 x1874663333300992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.955020:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333300992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.955028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.955031:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.955035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333300992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829313.955054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829313.955075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.955086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78a9880 x1874663333301120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.955097:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333301120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.955106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.955109:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.955113:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829313.955120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829313.955139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.955150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78aad80 x1874663333301248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.955170:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333301248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.955178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.955181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.955186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829313.955193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829313.955214:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829313.955265:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff9057c78a8000 x1874663333301376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829313.955279:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333301376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829313.955287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829313.955290:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829313.955294:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333301376:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829314.338392:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829315.362416:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829316.386440:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829317.410411:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829318.434472:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829319.074344:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829319.074358:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074380:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074388:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829319.074434:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333301504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.074457:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074460:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.074462:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074465:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074468:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074474:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829319.074483:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.074494:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333301632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.074506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829319.074515:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333301504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829319.074517:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074532:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074538:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074543:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829319.074548:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.074560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333301760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.074569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.074574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074582:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074599:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829319.074607:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829319.074611:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829319.074633:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333301888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.074646:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.074651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074657:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074663:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829319.074667:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78abb80 x1874663333301504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829319.074673:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829319.074690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829319.074703:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333301504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829319.074704:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333302016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829319.074743:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.074750:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829319.074752:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829319.074756:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333301504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829319.074768:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.074772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074778:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074781:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074785:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829319.074795:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829319.074795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829319.074827:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333302144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829319.074828:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.074840:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78aa680 x1874663333301632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829319.074844:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074862:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.074868:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.074876:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00100000:2.0:1787829319.074878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333301632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829319.074881:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074899:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829319.074901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829319.074906:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829319.074906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.074916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829319.074925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829319.074929:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333302272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.074945:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.074949:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.074950:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829319.074956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829319.074962:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11129 00000100:00080000:1.0:1787829319.074969:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.074976:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829319.074978:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78a9c00 x1874663333301760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829319.074984:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829319.075006:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333301760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829319.075007:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333302400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829319.075068:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075077:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829319.075086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829319.075108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.075115:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78aa300 x1874663333301888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829319.075128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333301888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829319.075133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075137:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333301888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829319.075141:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829319.075152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.075158:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78a9180 x1874663333302016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829319.075166:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333302016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829319.075172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075173:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829319.075187:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829319.075198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.075204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78a8700 x1874663333302144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829319.075212:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333302144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829319.075265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075269:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075272:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829319.075278:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829319.075296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.075304:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78a9500 x1874663333302272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829319.075312:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333302272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829319.075318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075321:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829319.075327:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829319.075337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829319.075343:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9057c78ab100 x1874663333302400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829319.075350:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333302400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829319.075356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829319.075357:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829319.075360:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333302400:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829319.458487:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829320.482422:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829321.506399:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829322.530456:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829323.554432:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829324.194353:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194362:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829324.194366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194375:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194386:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194393:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829324.194433:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333302528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194456:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194458:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829324.194460:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194463:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194475:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829324.194484:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333302656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194490:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829324.194494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194497:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194499:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194502:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829324.194509:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333302784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829324.194519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194524:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194526:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829324.194533:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333302528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829324.194546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333302912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829324.194571:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194573:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194576:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194579:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194582:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829324.194591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333303040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829324.194601:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194603:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194608:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829324.194612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829324.194621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333303168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829324.194633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.194636:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194638:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194640:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194644:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829324.194657:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb223800 x1874663333302528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.194658:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333303296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829324.194692:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829324.194695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829324.194696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829324.194700:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333302528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.194701:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11134 00000100:00080000:3.0:1787829324.194706:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.194712:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829324.194717:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829324.194719:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194724:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829324.194727:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333303424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829324.194731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333302528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829324.194744:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829324.194764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb220000 x1874663333302656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333302656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194786:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194788:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829324.194793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829324.194805:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb220a80 x1874663333302784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333302784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194823:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194827:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829324.194847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829324.194858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194865:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb223b80 x1874663333302912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194872:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333302912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194878:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333302912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829324.194884:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829324.194893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194899:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb221180 x1874663333303040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333303040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194919:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829324.194922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829324.194932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb222680 x1874663333303168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333303168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194951:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829324.194956:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829324.194965:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.194971:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb220e00 x1874663333303296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.194977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333303296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.194986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.194988:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.194990:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829324.194993:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829324.195002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829324.195008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9057fb221500 x1874663333303424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829324.195014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333303424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829324.195019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829324.195021:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829324.195023:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333303424:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829324.578469:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829325.602451:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829326.626514:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829327.650424:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829328.674362:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829329.314345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829329.314357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314383:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829329.314422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333303552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314446:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314449:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829329.314450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314453:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314456:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314458:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829329.314466:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829329.314476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333303680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829329.314488:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314490:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314492:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314495:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829329.314499:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829329.314506:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333303808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314528:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314534:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829329.314535:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333303552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829329.314536:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314546:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314551:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314556:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829329.314562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829329.314578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333303936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829329.314590:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314593:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314600:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829329.314604:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829329.314605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829329.314618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333304064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314629:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829329.314636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829329.314641:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314646:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314652:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829329.314658:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78a8a80 x1874663333303552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829329.314660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829329.314688:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333304192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829329.314689:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333303552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829329.314713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829329.314718:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829329.314720:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829329.314728:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829329.314729:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314736:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829329.314737:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333303552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829329.314742:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829329.314747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303680:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829329.314748:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829329.314764:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333304320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829329.314773:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829329.314775:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829329.314776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829329.314780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829329.314784:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11140 00000100:00080000:1.0:1787829329.314789:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829329.314792:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829329.314796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829329.314800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78ab100 x1874663333303680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829329.314807:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333304448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829329.314816:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333303680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.314842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.314846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.314851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829329.314877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829329.314896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.314905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78a9500 x1874663333303808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.314914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333303808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.314923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.314926:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.314930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829329.314936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829329.314956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.314969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78a8700 x1874663333303936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.314979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333303936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.314985:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.314987:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.314989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333303936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829329.314993:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829329.315007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.315014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78a9180 x1874663333304064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.315021:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333304064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.315026:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.315027:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.315029:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829329.315032:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829329.315042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.315048:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78aa300 x1874663333304192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.315060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333304192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.315071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.315073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.315075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829329.315078:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829329.315089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.315096:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78a9c00 x1874663333304320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.315103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333304320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.315108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.315110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.315111:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829329.315115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829329.315128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829329.315138:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829329/real 1787829329] req@ffff9057c78aa680 x1874663333304448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829329.315150:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333304448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829329.315165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829329.315169:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829329.315173:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333304448:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829329.698377:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829330.722416:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829331.746378:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829332.770418:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829333.794526:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829334.434401:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434410:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829334.434414:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434424:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434440:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434448:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829334.434502:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333304576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434526:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434529:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829334.434531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434534:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434548:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829334.434559:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333304704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434568:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434571:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829334.434572:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434574:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434577:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434579:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829334.434588:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333304832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434596:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434601:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829334.434603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829334.434603:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333304576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829334.434610:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434617:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434622:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829334.434635:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333304960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434644:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829334.434648:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434651:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434654:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434657:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434660:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829334.434671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333305088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434677:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829334.434682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434684:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434687:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434690:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434693:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829334.434696:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829334.434708:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333305216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434719:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434722:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829334.434723:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434727:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434731:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829334.434735:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434740:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829334.434745:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158c000 x1874663333304576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829334.434751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333305344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829334.434788:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829334.434791:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829334.434792:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829334.434797:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11145 00000100:00080000:0.0:1787829334.434799:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829334.434800:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333304576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829334.434803:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829334.434809:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829334.434821:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333305472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829334.434830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.434839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.434846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333304576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829334.434859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829334.434886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.434898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158f800 x1874663333304704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.434907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333304704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.434913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.434916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.434919:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829334.434957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829334.434979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.434989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158e680 x1874663333304832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.434997:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333304832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435003:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435005:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829334.435013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829334.435026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.435033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158df80 x1874663333304960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.435040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333304960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333304960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829334.435052:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829334.435063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.435069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158ca80 x1874663333305088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.435076:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333305088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435086:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829334.435095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829334.435115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.435127:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158f100 x1874663333305216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.435140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333305216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435150:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829334.435154:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829334.435168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.435176:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158e300 x1874663333305344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.435183:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333305344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435192:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829334.435204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829334.435217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829334.435272:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829334/real 1787829334] req@ffff9057f158dc00 x1874663333305472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829334.435283:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333305472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829334.435289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829334.435292:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829334.435294:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333305472:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829334.818420:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829335.842432:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829336.866433:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829337.890440:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829338.914457:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829339.554335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554345:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829339.554350:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554360:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554370:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554377:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829339.554416:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333305600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829339.554439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554441:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829339.554457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554462:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829339.554488:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333305600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829339.554492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333305728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829339.554502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554504:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829339.554506:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554508:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554511:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554517:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829339.554527:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333305856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829339.554533:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829339.554537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554539:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554542:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554544:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554548:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829339.554555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333305984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829339.554561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829339.554567:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554570:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829339.554572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554575:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554578:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554581:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829339.554592:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333306112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829339.554598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829339.554604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829339.554604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78abb80 x1874663333305600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829339.554611:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554616:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829339.554630:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333305600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829339.554635:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333306240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829339.554643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829339.554645:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554649:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829339.554650:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829339.554651:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829339.554656:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333305600:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829339.554660:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554665:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829339.554668:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554670:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829339.554677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333306368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829339.554682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305728:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829339.554691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829339.554695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829339.554696:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829339.554700:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11150 00000100:00080000:1.0:1787829339.554702:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829339.554707:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829339.554708:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829339.554715:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829339.554723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78a8000 x1874663333305728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829339.554726:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333306496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829339.554747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333305728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.554754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.554757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.554760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829339.554765:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829339.554777:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.554784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78aad80 x1874663333305856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.554791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333305856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.554796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.554798:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.554800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829339.554803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829339.554815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.554820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78a9880 x1874663333305984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.554828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333305984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.554832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.554834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.554837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333305984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829339.554843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829339.554860:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.554871:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78aaa00 x1874663333306112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.554884:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333306112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.554893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.554896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.554898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829339.554903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829339.554917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.554925:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78ab480 x1874663333306240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.554937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333306240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.554944:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.554947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.554951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829339.554969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829339.554990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.555000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78ab800 x1874663333306368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.555008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333306368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.555014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.555016:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.555017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829339.555021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829339.555033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829339.555039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829339/real 1787829339] req@ffff9057c78a8380 x1874663333306496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829339.555046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333306496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829339.555055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829339.555057:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829339.555059:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333306496:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829339.938387:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829340.962568:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829341.986445:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829343.010378:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829344.034393:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829344.674346:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674358:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829344.674363:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674374:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674386:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674394:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829344.674447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333306624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674471:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674474:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829344.674476:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674483:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674486:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674489:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829344.674497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829344.674507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333306752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829344.674518:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674522:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674526:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674531:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333306624:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829344.674531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829344.674541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333306880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674549:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674551:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829344.674552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674555:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674557:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674559:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829344.674562:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829344.674571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333307008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829344.674582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674584:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674599:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829344.674602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829344.674612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333307136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674618:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829344.674621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829344.674621:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829344.674624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674628:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674631:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674633:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829344.674637:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829344.674644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333307264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829344.674650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829344.674653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829344.674657:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674659:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674662:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829344.674665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829344.674674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333307392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829344.674681:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb222a00 x1874663333306624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829344.674712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829344.674715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829344.674716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829344.674721:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333306624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829344.674726:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11155 00000100:00080000:3.0:1787829344.674732:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829344.674735:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829344.674737:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829344.674739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829344.674743:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333306624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829344.674750:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333307520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829344.674763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829344.674786:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674796:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb220700 x1874663333306752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.674804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333306752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.674810:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.674812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829344.674842:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829344.674855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674862:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb222300 x1874663333306880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.674870:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333306880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.674876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.674878:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333306880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829344.674883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829344.674894:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb220380 x1874663333307008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.674908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333307008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.674913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.674915:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674917:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829344.674920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829344.674936:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb221880 x1874663333307136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.674952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333307136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.674958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.674960:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829344.674965:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829344.674976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.674982:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb223100 x1874663333307264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.674990:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333307264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.674995:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.674996:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.674998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829344.675002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829344.675011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.675017:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb222d80 x1874663333307392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.675024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333307392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.675030:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.675032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.675034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829344.675038:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829344.675048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829344.675053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff9057fb221c00 x1874663333307520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829344.675059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333307520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829344.675065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829344.675067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829344.675069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333307520:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829345.058436:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829346.082420:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829347.106429:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829348.130419:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829349.154391:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829349.794343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794353:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829349.794358:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794369:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794399:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794409:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829349.794455:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333307648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794488:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794492:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829349.794495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794509:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829349.794519:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829349.794534:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333307776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829349.794550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829349.794552:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333307648:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829349.794562:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794566:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794569:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829349.794573:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829349.794585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333307904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794595:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794599:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829349.794601:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794605:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794609:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794615:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829349.794620:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829349.794628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829349.794641:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333308032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794654:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829349.794657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794662:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794666:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794672:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829349.794683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78a9f80 x1874663333307648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829349.794690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829349.794711:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333307648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829349.794712:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333308160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794747:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794750:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829349.794750:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829349.794753:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794762:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:2.0:1787829349.794762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829349.794765:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794772:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829349.794773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333307648:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787829349.794776:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829349.794790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333308288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829349.794796:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794799:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829349.794800:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794802:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00080000:1.0:1787829349.794805:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829349.794809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307776:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829349.794811:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829349.794819:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829349.794836:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333308416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829349.794847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829349.794851:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829349.794857:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829349.794859:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829349.794866:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11160 00000100:00100000:2.0:1787829349.794866:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78a8380 x1874663333307776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829349.794872:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829349.794882:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829349.794890:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829349.794890:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333307776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.794904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829349.794908:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333308544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829349.794911:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.794920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829349.794933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829349.794952:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.794961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78ab800 x1874663333307904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.794970:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333307904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.794976:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.794978:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.794981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333307904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829349.794984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829349.794996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.795002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78ab480 x1874663333308032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.795010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333308032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.795015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.795017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.795019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829349.795022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829349.795032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.795038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78aaa00 x1874663333308160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.795045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333308160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.795050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.795052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.795054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829349.795057:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829349.795067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.795072:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78a9880 x1874663333308288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.795084:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333308288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.795089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.795091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.795092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829349.795115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829349.795136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.795148:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78aad80 x1874663333308416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.795160:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333308416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.795168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.795172:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.795175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829349.795182:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829349.795198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829349.795206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829349] req@ffff9057c78a8000 x1874663333308544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829349.795214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333308544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829349.795244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829349.795247:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829349.795250:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333308544:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829350.178390:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829351.202443:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829352.226452:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829353.250455:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829354.274367:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829354.914322:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914334:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829354.914348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914359:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914371:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914379:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829354.914420:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333308672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829354.914450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829354.914456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914461:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914466:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914471:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914482:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829354.914498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333308800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829354.914506:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333308672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829354.914531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829354.914537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914546:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914550:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914555:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829354.914567:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333308928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829354.914574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914576:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829354.914577:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914582:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914585:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914588:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829354.914596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333309056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829354.914605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829354.914615:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914618:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829354.914619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914623:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914636:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914641:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914644:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829354.914656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333309184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829354.914664:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914668:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829354.914670:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78abb80 x1874663333308672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829354.914671:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914694:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914705:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829354.914709:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333308672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829354.914712:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829354.914732:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.914738:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829354.914739:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333309312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829354.914749:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333308672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829354.914770:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914775:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829354.914776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914781:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914783:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.914786:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914789:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829354.914797:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333309440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829354.914811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308800:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829354.914813:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829354.914829:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829354.914833:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.914840:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11165 00000100:00080000:1.0:1787829354.914853:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829354.914860:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829354.914865:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829354.914873:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829354.914882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78aa680 x1874663333308800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829354.914895:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333309568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829354.914905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333308800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.914917:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.914921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.914925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829354.914931:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829354.914950:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.914959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78a9c00 x1874663333308928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.914966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333308928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.914971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.914973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.914975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333308928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829354.914979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829354.914990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.914997:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78aa300 x1874663333309056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.915004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333309056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.915009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.915010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.915012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829354.915016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829354.915026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.915033:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78a9180 x1874663333309184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.915040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333309184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.915045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.915047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.915049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829354.915053:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829354.915063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.915069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78a8700 x1874663333309312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.915075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333309312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.915080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.915082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.915084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829354.915097:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829354.915108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.915114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78a9500 x1874663333309440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.915121:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333309440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.915127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.915129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.915131:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829354.915134:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829354.915144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829354.915150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff9057c78ab100 x1874663333309568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829354.915157:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333309568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829354.915162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829354.915164:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829354.915166:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333309568:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829355.298538:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829356.322440:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829357.346441:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829358.370438:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829359.394456:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829360.034336:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034345:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829360.034349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034359:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034376:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829360.034415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333309696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034439:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829360.034443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034446:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034448:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034451:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829360.034470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333309824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034480:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829360.034481:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034483:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034485:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034487:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829360.034496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333309696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829360.034499:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333309952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034510:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034513:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829360.034514:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034517:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034519:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034521:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829360.034531:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333310080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034538:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034540:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829360.034542:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034545:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034548:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034550:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829360.034560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333310208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034566:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034568:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829360.034569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829360.034575:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829360.034577:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034581:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034584:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034588:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829360.034596:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333310336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829360.034601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829360.034604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034607:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034610:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034613:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034616:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829360.034630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333310464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829360.034630:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78a8a80 x1874663333309696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829360.034646:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829360.034652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829360.034655:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829360.034661:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11170 00000100:00080000:1.0:1787829360.034663:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829360.034666:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829360.034670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829360.034675:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333309696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829360.034679:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333310592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829360.034701:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.034707:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.034713:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333309696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829360.034722:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829360.034746:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.034759:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78ab100 x1874663333309824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.034773:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333309824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.034782:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.034785:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.034789:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829360.034811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829360.034826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.034836:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78a9500 x1874663333309952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.034847:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333309952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.034856:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.034860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.034863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333309952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829360.034869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829360.034887:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.034898:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78a8700 x1874663333310080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.034911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333310080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.034920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.034923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.034926:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829360.034932:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829360.034953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.034964:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78a9180 x1874663333310208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.034976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333310208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.034983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.034986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.034989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829360.034995:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829360.035016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.035028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78aa300 x1874663333310336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.035047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333310336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.035056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.035059:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.035063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829360.035069:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829360.035089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.035100:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78a9c00 x1874663333310464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.035112:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333310464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.035121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.035124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.035127:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829360.035132:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829360.035152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829360.035163:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9057c78aa680 x1874663333310592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829360.035175:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333310592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829360.035189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829360.035193:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829360.035196:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333310592:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829360.418374:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829361.442436:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829362.466587:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829363.490438:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829364.514434:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829365.154308:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154317:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829365.154322:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154331:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154340:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154347:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829365.154384:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333310720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829365.154407:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154410:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829365.154411:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154415:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154417:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154421:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154429:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829365.154438:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333310848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829365.154447:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154451:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829365.154453:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829365.154453:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333310720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829365.154458:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154460:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154465:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829365.154486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333310976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829365.154497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154501:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829365.154503:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154508:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154513:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154518:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154523:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829365.154528:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829365.154536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333311104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829365.154549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154552:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829365.154554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154558:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154562:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829365.154565:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78abb80 x1874663333310720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829365.154568:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154575:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829365.154586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333311232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829365.154589:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333310720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829365.154605:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154609:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829365.154610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829365.154611:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829365.154615:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829365.154618:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154622:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829365.154623:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333310720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829365.154630:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154633:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829365.154641:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333311360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829365.154648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829365.154650:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829365.154657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154661:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154664:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154667:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829365.154671:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154673:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829365.154683:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333311488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829365.154686:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78a8000 x1874663333310848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829365.154705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829365.154710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829365.154710:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333310848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829365.154714:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829365.154721:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11175 00000100:00080000:1.0:1787829365.154723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829365.154725:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829365.154728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829365.154739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829365.154740:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333311616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829365.154749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.154758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829365.154767:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829365.154791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.154801:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78aad80 x1874663333310976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.154810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333310976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.154815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.154817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.154820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333310976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829365.154824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829365.154838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.154844:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78a9880 x1874663333311104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.154851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333311104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.154856:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.154858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.154860:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829365.154863:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829365.154873:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.154882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78aaa00 x1874663333311232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.154895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333311232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.154904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.154907:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.154911:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829365.154929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829365.154950:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.154959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78ab480 x1874663333311360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.154968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333311360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.154973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.154976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.154978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829365.154982:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829365.154994:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.155001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78ab800 x1874663333311488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.155008:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333311488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.155013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.155015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.155017:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829365.155021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829365.155031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829365.155036:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829365] req@ffff9057c78a8380 x1874663333311616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829365.155043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333311616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829365.155048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829365.155050:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829365.155052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333311616:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829365.538391:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829366.562409:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829367.586384:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829368.610339:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829369.634529:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829370.274337:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829370.274350:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274376:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829370.274414:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333311744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829370.274441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274445:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829370.274447:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274453:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274457:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274461:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829370.274486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333311872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829370.274493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333311744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829370.274497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274501:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829370.274502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274506:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274513:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274517:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829370.274526:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333312000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829370.274535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829370.274539:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274543:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274545:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274553:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829370.274559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829370.274562:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333312128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829370.274570:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829370.274575:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274578:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274582:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274585:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274590:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829370.274599:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333312256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829370.274603:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158d880 x1874663333311744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829370.274609:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274612:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829370.274614:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829370.274618:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829370.274622:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333311744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829370.274625:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274629:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829370.274640:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333312384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829370.274646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829370.274648:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829370.274651:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829370.274652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829370.274654:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829370.274656:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333311744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829370.274658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274662:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829370.274670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829370.274681:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829370.274685:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333312512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829370.274695:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829370.274699:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829370.274702:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829370.274703:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829370.274706:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11181 00000100:00080000:0.0:1787829370.274709:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829370.274713:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274716:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158ce00 x1874663333311872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829370.274718:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829370.274728:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333312640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829370.274729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333311872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274742:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274745:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333311872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.274752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829370.274772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158c700 x1874663333312000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333312000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274801:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.274810:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829370.274825:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274832:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158ea00 x1874663333312128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333312128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829370.274852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829370.274862:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158d180 x1874663333312256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274875:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333312256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274882:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274884:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829370.274887:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829370.274897:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274903:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158d500 x1874663333312384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333312384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274923:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.274936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829370.274947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158ed80 x1874663333312512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274959:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333312512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.274964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.274965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.274967:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.274970:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829370.274980:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829370.274985:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829370/real 1787829370] req@ffff9057f158fb80 x1874663333312640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.274992:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333312640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.275001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.275003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829370.275005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333312640:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829370.658364:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829371.682426:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829372.706419:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829373.730418:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829374.754438:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829375.394302:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394311:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829375.394315:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394327:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394336:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394343:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829375.394381:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333312768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394404:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394407:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829375.394409:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394413:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394415:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394418:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394426:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829375.394436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333312896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394444:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829375.394450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829375.394452:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333312768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829375.394456:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394467:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829375.394474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333313024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394480:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394483:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829375.394489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394492:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394495:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394498:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394502:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829375.394510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333313152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394517:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394520:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829375.394521:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394524:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394526:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394529:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394532:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829375.394545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333313280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394555:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829375.394563:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312896:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829375.394565:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394575:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394581:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394587:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829375.394596:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829375.394609:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333313408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829375.394617:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313152:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829375.394620:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394624:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829375.394626:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394631:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394635:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829375.394639:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313280:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829375.394641:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829375.394649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829375.394659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333313536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829375.394666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829375.394669:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829375.394671:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829375.394674:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11186 00000100:00080000:1.0:1787829375.394677:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829375.394682:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829375.394683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313408:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829375.394686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829375.394698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333313664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829375.394720:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829375.394748:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313664:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829375.778399:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829376.802453:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829377.826387:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829378.466499:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787829378.466540:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829378.466560:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466608:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466619:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466629:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466641:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466652:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829378.466654:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78a8380 x1874663333312896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829378.466664:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829378.466679:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829378.466697:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333312896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466714:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466722:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466728:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333312896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829378.466737:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78ab800 x1874663333313024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466750:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333313024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466759:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466770:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829378.466777:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78ab480 x1874663333313152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466789:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333313152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829378.466812:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78aaa00 x1874663333313280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333313280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466837:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829378.466847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78a9f80 x1874663333312768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333312768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333312768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829378.466918:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78a8000 x1874663333313664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333313664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466941:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466948:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829378.466956:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78a9880 x1874663333313408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.466969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333313408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.466978:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.466982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.466985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829378.466993:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829375/real 1787829378] req@ffff9057c78aad80 x1874663333313536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829378.467006:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333313536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829378.467014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829378.467018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829378.467022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333313536:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1787829378.850446:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829379.874412:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829380.514330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514344:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829380.514348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514361:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514374:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514381:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829380.514414:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333313792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514443:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514446:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829380.514448:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514451:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514453:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514456:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829380.514464:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829380.514473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333313920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514480:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514483:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829380.514484:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514488:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514492:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829380.514498:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829380.514498:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333313792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829380.514510:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333314048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514522:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829380.514528:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514533:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514538:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514543:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829380.514549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829380.514562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333314176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514571:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829380.514574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514582:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829380.514590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829380.514596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829380.514611:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333314304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829380.514625:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829380.514627:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829380.514632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514636:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514641:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829380.514647:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829380.514654:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78abb80 x1874663333313792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829380.514664:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333314432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829380.514681:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333313792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829380.514699:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829380.514703:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829380.514705:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829380.514711:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829380.514715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829380.514721:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00100000:2.0:1787829380.514726:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333313792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829380.514729:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514739:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829380.514743:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313920:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829380.514748:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829380.514767:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333314560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829380.514779:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829380.514781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829380.514784:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829380.514802:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829380.514802:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78aa680 x1874663333313920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829380.514812:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11191 00000100:00080000:1.0:1787829380.514820:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829380.514827:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829380.514832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333313920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829380.514835:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829380.514850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829380.514856:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333314688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829380.514859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.514868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333313920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829380.514906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829380.514933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.514947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78a9c00 x1874663333314048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.514960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333314048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.514969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.514973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.514977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829380.514984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829380.515007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.515020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78aa300 x1874663333314176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.515033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333314176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.515042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.515046:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.515049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829380.515056:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829380.515078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.515091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78a9180 x1874663333314304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.515105:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333314304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.515113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.515118:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.515122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829380.515128:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829380.515150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.515181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78a8700 x1874663333314432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.515196:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333314432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.515205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.515208:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.515213:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829380.515256:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829380.515280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.515293:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78a9500 x1874663333314560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.515304:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333314560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.515313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.515316:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.515320:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829380.515327:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829380.515347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829380.515358:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829380/real 1787829380] req@ffff9057c78ab100 x1874663333314688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829380.515370:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333314688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829380.515378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829380.515381:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829380.515384:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333314688:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829380.898394:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829381.922431:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829382.946460:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829383.970519:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829384.994359:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829385.634371:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634385:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829385.634390:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634403:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634417:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.634426:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829385.634473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333314816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829385.634509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634518:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.634523:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634533:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829385.634551:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333314944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634569:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829385.634571:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634577:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00100000:2.0:1787829385.634585:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333314816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829385.634591:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.634598:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634604:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829385.634624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333315072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634634:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829385.634640:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634644:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634649:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.634654:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829385.634662:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829385.634683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333315200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634702:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829385.634707:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158fb80 x1874663333314816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829385.634710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634727:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634734:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829385.634738:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333314816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829385.634742:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829385.634761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.634769:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.634787:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333314816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829385.634789:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333315328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634807:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.634811:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829385.634814:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634819:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634823:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829385.634828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314944:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829385.634834:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634846:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829385.634860:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333315456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.634873:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829385.634876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829385.634883:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829385.634888:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634894:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00080000:0.0:1787829385.634900:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829385.634902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158ed80 x1874663333314944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829385.634908:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.634917:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829385.634933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333314944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829385.634941:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333315584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829385.634958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.634970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829385.634971:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829385.634979:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333314944:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829385.634981:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829385.634988:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.634998:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11196 00000100:00100000:2.0:1787829385.634998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315072:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829385.635008:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.635015:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829385.635021:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829385.635034:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829385.635039:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333315712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829385.635056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158d500 x1874663333315072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333315072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635092:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635095:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635099:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829385.635106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829385.635142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829385.635156:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158d180 x1874663333315200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635169:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333315200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635185:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829385.635191:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829385.635211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829385.635255:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158ea00 x1874663333315328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635269:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333315328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635289:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635293:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829385.635314:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829385.635337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829385.635351:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158c700 x1874663333315456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635363:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333315456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635377:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635380:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829385.635387:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829385.635407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829385.635419:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158ce00 x1874663333315584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635433:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333315584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635445:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829385.635456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829385.635476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829385.635487:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff9057f158d880 x1874663333315712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829385.635500:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333315712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829385.635508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829385.635512:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829385.635516:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333315712:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829386.018437:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829387.042343:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829388.066414:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829389.090363:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829390.114421:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829390.754370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754381:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829390.754387:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754398:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754411:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754420:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829390.754464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333315840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829390.754497:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829390.754504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754509:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754512:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754516:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829390.754525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829390.754538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333315968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829390.754544:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333315840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829390.754549:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829390.754557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754560:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754563:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754565:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829390.754568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829390.754578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333316096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829390.754584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829390.754589:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754592:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754596:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754598:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829390.754602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829390.754608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333316224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829390.754614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829390.754618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754621:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754624:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829390.754626:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829390.754630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829390.754634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829390.754643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333316352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829390.754650:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829390.754653:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754657:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754660:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754663:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829390.754667:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829390.754672:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78a8a80 x1874663333315840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.754677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333316480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829390.754693:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754695:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829390.754700:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829390.754700:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333315840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.754709:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754714:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754718:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829390.754720:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829390.754721:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829390.754727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829390.754732:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333316608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829390.754733:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333315840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829390.754748:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829390.754751:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829390.754752:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829390.754755:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11201 00000100:00080000:1.0:1787829390.754757:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829390.754760:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829390.754763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315968:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829390.754765:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829390.754778:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333316736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829390.754789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.754800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78ab100 x1874663333315968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.754810:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333315968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.754819:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.754823:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.754826:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333315968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829390.754833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829390.754856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.754869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78a9500 x1874663333316096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.754882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333316096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.754891:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.754894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.754898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829390.754905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829390.754926:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.754938:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78a8700 x1874663333316224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.754958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333316224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.754967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.754970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.754974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829390.754980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829390.755002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.755013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78a9180 x1874663333316352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.755026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333316352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.755036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.755039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.755043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829390.755049:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829390.755069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.755081:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78aa300 x1874663333316480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.755111:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333316480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.755126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.755129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.755133:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829390.755140:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829390.755160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.755172:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78a9c00 x1874663333316608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.755185:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333316608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.755194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.755198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.755201:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829390.755268:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829390.755291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829390.755304:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff9057c78aa680 x1874663333316736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829390.755317:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333316736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829390.755326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829390.755329:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829390.755333:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333316736:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829391.138478:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829392.162390:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829393.186363:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829394.210419:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829395.234388:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829395.874353:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874362:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829395.874366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874377:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874386:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874393:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829395.874429:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333316864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829395.874460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829395.874466:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874476:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874492:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829395.874511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333316992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829395.874512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333316864:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829395.874530:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829395.874537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874547:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874552:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829395.874570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333317120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829395.874580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829395.874581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829395.874587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874592:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874596:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874600:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874604:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829395.874619:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333317248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829395.874622:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78abb80 x1874663333316864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.874652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829395.874657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333316864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.874659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829395.874662:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874671:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874676:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874679:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829395.874689:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.874698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829395.874699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333317376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829395.874703:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333316864:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829395.874720:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874725:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829395.874727:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829395.874731:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316992:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829395.874736:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874742:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874746:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874751:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829395.874756:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829395.874766:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333317504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829395.874775:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829395.874777:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78aad80 x1874663333316992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.874782:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829395.874786:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.874796:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00100000:2.0:1787829395.874801:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333316992/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.874802:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829395.874812:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829395.874813:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.874821:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829395.874823:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829395.874824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333316992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829395.874834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829395.874844:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333317632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829395.874856:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829395.874858:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829395.874861:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829395.874863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829395.874867:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11206 00000100:00080000:1.0:1787829395.874873:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829395.874876:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78a9880 x1874663333317120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.874880:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829395.874890:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829395.874896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333317120/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829395.874902:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333317760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829395.874908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.874913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.874916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829395.874925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829395.874948:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829395.874957:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78a8000 x1874663333317248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.874967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333317248/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.874974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.874977:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.874980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829395.874984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829395.874994:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829395.875000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78a9f80 x1874663333317376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.875013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333317376/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.875018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.875020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.875022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333317376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829395.875033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829395.875043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829395.875050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78aaa00 x1874663333317504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.875074:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333317504/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.875080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.875082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.875084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829395.875088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829395.875104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829395.875116:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78ab480 x1874663333317632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.875129:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333317632/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.875144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.875148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.875152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829395.875159:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829395.875195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829395.875204:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff9057c78ab800 x1874663333317760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.875214:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333317760/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.875249:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.875253:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829395.875255:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333317760:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829396.258427:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829397.282383:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829398.306500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829399.330379:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829400.354432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829400.994311:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994319:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829400.994330:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994340:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994350:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994357:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829400.994396:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333317888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994419:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994422:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829400.994423:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994427:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994429:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994432:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994440:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829400.994450:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333318016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994457:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994459:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829400.994460:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994462:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994464:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829400.994480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333318144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994495:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829400.994497:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829400.994503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333317888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829400.994504:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994520:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994528:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829400.994546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333318272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829400.994563:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994567:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994571:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829400.994576:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829400.994577:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829400.994593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333318400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829400.994604:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994608:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994610:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994613:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829400.994628:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78a8380 x1874663333317888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829400.994632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333318528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829400.994663:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829400.994666:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829400.994667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994671:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00080000:1.0:1787829400.994673:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994678:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994678:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333317888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829400.994684:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829400.994695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333318656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829400.994697:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.994706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829400.994709:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829400.994713:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333317888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829400.994714:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829400.994718:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829400.994722:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11211 00000100:00100000:2.0:1787829400.994725:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318016:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829400.994726:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829400.994733:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829400.994737:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829400.994747:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333318784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829400.994756:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78ab800 x1874663333318016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.994781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333318016/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.994787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.994790:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.994792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829400.994796:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829400.994807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78ab480 x1874663333318144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.994821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333318144/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.994826:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.994828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.994830:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829400.994849:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829400.994861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78aaa00 x1874663333318272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.994875:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333318272/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.994882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.994885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.994889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829400.994895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829400.994914:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78a9f80 x1874663333318400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.994937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333318400/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.994945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.994949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.994953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333318400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829400.994959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829400.994976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.994985:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78a8000 x1874663333318528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.994998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333318528/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.995004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.995006:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.995008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829400.995012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829400.995042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.995049:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78a9880 x1874663333318656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.995057:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333318656/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.995062:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.995064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.995066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829400.995070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829400.995081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829400.995086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff9057c78aad80 x1874663333318784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.995093:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333318784/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829400.995103:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829400.995105:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829400.995107:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333318784:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829401.378420:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829402.402408:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829403.426502:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829404.450391:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829405.474500:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829406.114365:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114374:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829406.114379:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114389:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114399:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114406:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829406.114443:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333318912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114470:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.114472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114477:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.114496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333319040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.114507:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114510:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114512:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114514:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114518:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.114526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333319168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829406.114539:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333318912:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829406.114541:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114547:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.114551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114554:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114556:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114558:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829406.114569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333319296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.114579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114582:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114583:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114586:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114588:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829406.114595:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333319424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114601:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114604:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.114605:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114608:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114611:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114614:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114617:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829406.114623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829406.114627:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333319552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114634:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114637:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.114639:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114642:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00080000:0.0:1787829406.114644:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114646:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.114656:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333319680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.114666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.114669:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.114670:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.114676:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11216 00000100:00100000:2.0:1787829406.114677:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158dc00 x1874663333318912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829406.114681:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.114689:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829406.114698:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829406.114709:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333318912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829406.114714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333319808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829406.114745:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.114752:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.114758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333318912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829406.114768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829406.114795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.114809:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158e300 x1874663333319040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.114824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333319040/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.114833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.114836:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.114841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829406.114868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829406.114889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.114902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158f100 x1874663333319168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.114915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333319168/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.114923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.114926:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.114930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829406.114936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829406.114957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.114969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158ca80 x1874663333319296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.114999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333319296/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.115008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.115011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.115015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829406.115021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829406.115043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.115056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158df80 x1874663333319424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.115068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333319424/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.115077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.115081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.115084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333319424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829406.115090:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829406.115110:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.115122:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158e680 x1874663333319552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.115134:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333319552/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.115143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.115146:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.115150:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829406.115156:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829406.115175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.115186:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158f800 x1874663333319680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.115198:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333319680/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.115208:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.115211:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.115215:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829406.115250:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829406.115273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829406.115285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9057f158c000 x1874663333319808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829406.115298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333319808/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829406.115307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829406.115311:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829406.115314:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333319808:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829406.498400:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829407.522423:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829408.546393:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829409.570418:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829410.594384:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829411.234284:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234300:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829411.234303:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234313:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234324:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234330:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829411.234375:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333319936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234405:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829411.234410:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234413:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234417:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234420:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234430:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829411.234444:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333320064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234452:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829411.234456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234461:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234467:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829411.234474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333320192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234482:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234485:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829411.234491:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829411.234492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333319936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829411.234505:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234511:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234516:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234523:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829411.234542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333320320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829411.234559:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234562:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234565:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234569:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829411.234583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829411.234594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333320448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829411.234609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234614:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234617:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234621:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234624:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829411.234632:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333320576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829411.234644:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829411.234650:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158c000 x1874663333319936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829411.234651:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829411.234656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829411.234670:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:0.0:1787829411.234677:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829411.234680:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234693:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829411.234706:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333320704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829411.234712:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333319936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829411.234729:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829411.234737:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829411.234740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829411.234741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829411.234746:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11221 00000100:00080000:2.0:1787829411.234751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829411.234754:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829411.234760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333319936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829411.234764:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829411.234778:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829411.234782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829411.234791:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333320832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829411.234808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.234826:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158f800 x1874663333320064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.234835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333320064/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.234841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.234844:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.234846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829411.234869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829411.234882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.234888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158e680 x1874663333320192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.234896:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333320192/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.234901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.234903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.234905:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829411.234909:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829411.234921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.234928:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158df80 x1874663333320320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.234935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333320320/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.234940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.234942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.234957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829411.234962:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829411.234976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.234983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158ca80 x1874663333320448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.234990:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333320448/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.235003:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.235005:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.235008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333320448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829411.235012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829411.235024:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.235031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158f100 x1874663333320576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.235044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333320576/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.235049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.235051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.235053:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829411.235056:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829411.235067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.235073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158e300 x1874663333320704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.235081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333320704/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.235086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.235088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.235090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829411.235093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829411.235107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829411.235117:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829411/real 1787829411] req@ffff9057f158dc00 x1874663333320832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829411.235130:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333320832/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829411.235153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829411.235156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829411.235160:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333320832:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829411.618424:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829412.642463:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829413.666393:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829414.690423:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829415.714418:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829416.354325:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354335:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829416.354339:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354365:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829416.354401:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333320960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829416.354425:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354428:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829416.354430:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354435:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354437:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354444:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829416.354453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333321088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829416.354465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829416.354469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354473:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829416.354489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333321216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829416.354490:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333320960:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829416.354500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354503:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829416.354504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354506:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354508:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354510:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829416.354519:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333321344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829416.354525:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829416.354529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354532:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354536:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354541:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829416.354559:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333321472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829416.354568:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829416.354574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829416.354575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829416.354580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354586:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354591:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354596:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829416.354617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333321600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829416.354627:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829416.354631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158d880 x1874663333320960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829416.354632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829416.354637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829416.354646:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354650:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829416.354654:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829416.354660:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829416.354675:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333321728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829416.354684:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333320960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829416.354711:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829416.354719:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829416.354721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829416.354724:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829416.354727:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11227 00000100:00080000:0.0:1787829416.354731:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829416.354733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829416.354738:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829416.354744:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333320960:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787829416.354746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829416.354760:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829416.354762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333321856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829416.354795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.354814:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158ce00 x1874663333321088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.354827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333321088/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.354835:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.354839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.354844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829416.354870:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829416.354890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.354902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158c700 x1874663333321216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.354931:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333321216/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.354940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.354944:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.354948:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829416.354955:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829416.354975:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.354986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158ea00 x1874663333321344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.354999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333321344/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.355007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.355010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.355013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829416.355020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829416.355040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.355052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158d180 x1874663333321472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.355063:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333321472/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.355071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.355074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.355078:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333321472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829416.355084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829416.355103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.355114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158d500 x1874663333321600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.355126:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333321600/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.355134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.355137:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.355140:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829416.355146:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829416.355165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.355176:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158ed80 x1874663333321728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.355187:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333321728/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.355195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.355198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.355201:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829416.355208:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829416.355285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829416.355299:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829416/real 1787829416] req@ffff9057f158fb80 x1874663333321856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829416.355312:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333321856/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829416.355320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829416.355323:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829416.355326:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333321856:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829416.738371:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829417.762386:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829418.786387:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829419.810504:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829420.834422:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829421.474316:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474324:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829421.474328:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474337:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474347:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474354:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829421.474402:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333321984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474423:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474425:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829421.474426:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474430:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474432:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474435:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474443:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829421.474452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333322112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829421.474464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829421.474469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333321984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829421.474470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474476:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474480:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474484:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829421.474491:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333322240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474498:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829421.474501:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474504:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474507:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474510:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474514:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829421.474525:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333322368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829421.474541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474548:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11232 00000800:00000100:2.0:1787829421.474550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829421.474554:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474560:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829421.474578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333322496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474589:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829421.474591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474594:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474598:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474601:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829421.474606:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78abb80 x1874663333321984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829421.474617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333322624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829421.474624:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474629:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829421.474629:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333321984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829421.474633:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474641:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474646:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474650:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829421.474651:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829421.474653:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829421.474661:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333322752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829421.474664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474673:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333321984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829421.474677:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829421.474680:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829421.474682:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829421.474686:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11232 00000100:00080000:1.0:1787829421.474688:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829421.474691:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829421.474694:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829421.474698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829421.474703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333322880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829421.474724:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78aa680 x1874663333322112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474745:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333322112/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474754:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474756:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829421.474760:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829421.474771:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474778:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78a9c00 x1874663333322240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474785:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333322240/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474790:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474792:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474794:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829421.474797:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829421.474809:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474815:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78aa300 x1874663333322368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474822:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333322368/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474829:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474831:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829421.474834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829421.474844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78a9180 x1874663333322496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474863:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333322496/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474868:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474871:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474887:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333322496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829421.474892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829421.474907:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474914:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78a8700 x1874663333322624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333322624/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829421.474947:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829421.474959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.474965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78a9500 x1874663333322752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.474972:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333322752/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.474977:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.474979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.474981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829421.474994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829421.475006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829421.475012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829421/real 1787829421] req@ffff9057c78ab100 x1874663333322880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829421.475019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333322880/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829421.475029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829421.475031:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829421.475033:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333322880:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829421.858401:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829422.882499:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829423.906435:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829424.930437:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829425.954360:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829426.594335:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594354:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829426.594359:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594371:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829426.594430:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333323008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829426.594459:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594463:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594465:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594469:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829426.594493:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333323136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594502:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829426.594505:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333323008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829426.594506:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829426.594508:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594516:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594518:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594520:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829426.594533:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333323264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594540:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594543:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829426.594545:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594547:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594549:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594551:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594555:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829426.594562:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333323392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829426.594572:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594575:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594578:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594583:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829426.594583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594590:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829426.594598:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333323520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829426.594609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594612:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594614:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594617:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829426.594635:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333323648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829426.594635:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158fb80 x1874663333323008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829426.594658:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829426.594665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594668:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:0.0:1787829426.594671:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594674:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829426.594678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829426.594685:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333323008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829426.594686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333323776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.594717:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.594720:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829426.594721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829426.594722:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.594732:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11237 00000100:00080000:2.0:1787829426.594733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829426.594735:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.594740:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829426.594743:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333323008:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787829426.594745:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829426.594756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333323904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829426.594775:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829426.594798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.594810:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158ed80 x1874663333323136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.594823:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333323136/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.594832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.594835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.594853:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829426.594859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829426.594877:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.594887:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158d500 x1874663333323264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.594899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333323264/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.594907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.594910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.594913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829426.594919:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829426.594935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.594943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158d180 x1874663333323392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.594955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333323392/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.594964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.594966:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.594969:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829426.594975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829426.594989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.594998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158ea00 x1874663333323520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.595010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333323520/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.595019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.595022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.595025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333323520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829426.595030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829426.595045:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.595053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158c700 x1874663333323648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.595065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333323648/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.595073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.595075:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.595079:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829426.595084:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829426.595098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.595107:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158ce00 x1874663333323776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.595119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333323776/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.595128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.595131:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.595134:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829426.595140:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829426.595153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829426.595161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff9057f158d880 x1874663333323904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829426.595172:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333323904/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829426.595180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829426.595183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829426.595187:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333323904:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829426.978546:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829428.002404:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829429.026413:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829430.050378:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829431.074432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829431.714342:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714352:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829431.714357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714367:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714385:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829431.714434:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333324032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714462:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829431.714467:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714477:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829431.714498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333324160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714506:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714510:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829431.714511:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714516:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714519:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829431.714519:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333324032:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829431.714525:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714531:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829431.714539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333324288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829431.714550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714553:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714555:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714557:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829431.714568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333324416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714573:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714576:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829431.714578:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714580:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714583:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714585:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829431.714590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829431.714602:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333324544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829431.714613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714615:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714618:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714620:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829431.714634:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78a8a80 x1874663333324032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.714635:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333324672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829431.714657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714662:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829431.714664:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714668:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00100000:2.0:1787829431.714671:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333324032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.714672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714680:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829431.714690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829431.714696:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333324800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829431.714699:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714708:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333324032:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829431.714712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829431.714715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829431.714717:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829431.714720:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11242 00000100:00080000:1.0:1787829431.714723:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829431.714725:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829431.714728:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829431.714733:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829431.714738:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333324928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829431.714758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714771:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78ab100 x1874663333324160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333324160/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829431.714795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829431.714822:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78a9500 x1874663333324288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714837:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333324288/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714845:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829431.714851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829431.714865:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714870:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78a8700 x1874663333324416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333324416/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714884:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714886:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829431.714889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829431.714899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78a9180 x1874663333324544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333324544/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714917:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333324544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829431.714923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829431.714933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78aa300 x1874663333324672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333324672/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714957:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714959:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829431.714962:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829431.714972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.714977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78a9c00 x1874663333324800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.714984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333324800/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.714988:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.714990:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.714992:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829431.715005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829431.715015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829431.715021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff9057c78aa680 x1874663333324928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829431.715028:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333324928/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829431.715038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829431.715039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829431.715041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333324928:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829432.098407:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829433.122471:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829434.146657:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829435.170430:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829436.194395:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829436.834346:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834355:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829436.834359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834370:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834379:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829436.834386:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829436.834426:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333325056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829436.834453:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834456:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829436.834458:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834464:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829436.834466:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829436.834474:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829436.834482:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333325184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829436.834491:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829436.834495:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834497:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834499:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829436.834501:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829436.834504:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829436.834514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333325312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829436.834515:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333325056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829436.834531:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834533:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829436.834534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834537:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834541:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829436.834545:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829436.834551:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829436.834565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333325440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829436.834575:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834580:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829436.834582:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834587:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834592:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829436.834594:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829436.834599:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829436.834607:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829436.834621:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333325568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829436.834632:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834636:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829436.834638:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829436.834642:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834646:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829436.834652:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829436.834652:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78abb80 x1874663333325056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829436.834660:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829436.834677:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333325696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829436.834680:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333325056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.834699:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834702:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829436.834704:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829436.834704:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.834713:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829436.834714:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834720:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829436.834722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333325056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829436.834726:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829436.834732:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829436.834734:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829436.834742:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333325824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829436.834758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829436.834759:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829436.834778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829436.834782:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829436.834785:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78aad80 x1874663333325184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.834791:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11247 00000100:00080000:1.0:1787829436.834801:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829436.834806:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333325184/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.834812:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829436.834821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829436.834824:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829436.834826:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.834832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829436.834845:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333325952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829436.834864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829436.834882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.834892:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78a9880 x1874663333325312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.834901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333325312/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.834907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.834909:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.834911:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829436.834914:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829436.834925:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.834931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78a8000 x1874663333325440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.834938:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333325440/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.834942:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.834944:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.834946:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829436.834949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829436.834959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.834965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78a9f80 x1874663333325568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.834973:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333325568/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.834978:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.834980:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.834982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333325568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829436.834985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829436.834995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.835000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78aaa00 x1874663333325696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.835007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333325696/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.835012:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.835014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.835016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829436.835019:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829436.835028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.835034:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78ab480 x1874663333325824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.835041:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333325824/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.835046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.835048:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.835050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829436.835054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829436.835063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829436.835069:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff9057c78ab800 x1874663333325952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.835075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333325952/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.835080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829436.835082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829436.835084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333325952:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829437.218420:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829438.242385:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829439.266429:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829440.290402:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829441.314462:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829441.954289:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954300:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829441.954306:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954316:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954326:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954333:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829441.954374:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333326080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954407:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954411:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829441.954413:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954418:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954422:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954427:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954439:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829441.954457:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333326080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829441.954458:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333326208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954481:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829441.954483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954487:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954491:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954496:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829441.954517:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333326336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954527:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829441.954531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954534:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954537:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954540:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829441.954555:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333326464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954563:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954566:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829441.954568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954571:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954573:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954575:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829441.954584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829441.954592:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333326592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954600:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954602:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829441.954604:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954608:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954611:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954614:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954618:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829441.954626:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829441.954630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333326720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954640:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954643:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829441.954645:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954647:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954650:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829441.954652:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829441.954659:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829441.954667:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333326848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829441.954675:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829441.954680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829441.954682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829441.954688:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11252 00000100:00080000:1.0:1787829441.954692:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829441.954695:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326592:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829441.954701:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829441.954711:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829441.954729:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333326976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829441.954749:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829441.954783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829441.954809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326976:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829442.338423:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829443.362431:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829444.386408:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829445.026509:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829445.026551:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829445.026573:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026617:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026629:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026640:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026651:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026660:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829445.026671:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829445.026675:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78a9f80 x1874663333326592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829445.026687:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829445.026733:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333326592/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026748:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026755:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026760:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829445.026769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78a8000 x1874663333326720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026784:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333326720/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026794:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026803:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829445.026812:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78a9880 x1874663333326848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333326848/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026846:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829445.026855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78ab800 x1874663333326208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333326208/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026881:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026884:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829445.026891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78ab480 x1874663333326336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333326336/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026913:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026916:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829445.026923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78aaa00 x1874663333326464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026935:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333326464/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026946:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333326464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829445.026956:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78a8380 x1874663333326080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.026966:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333326080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.026973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.026976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.026979:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333326080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829445.027012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829445] req@ffff9057c78aad80 x1874663333326976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829445.027023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333326976/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829445.027031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829445.027035:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829445.027039:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333326976:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829445.410396:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829446.434426:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829447.074403:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074416:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829447.074422:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074434:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074446:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074456:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829447.074515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333327104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.074545:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.074549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074553:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074557:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074561:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074570:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.074582:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333327232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829447.074592:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333327104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829447.074593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074599:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.074603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074606:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074609:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074611:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074614:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.074623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333327360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.074629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.074634:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074638:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074643:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074647:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074653:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829447.074664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829447.074669:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333327488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.074681:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074684:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.074686:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074690:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074709:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074714:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829447.074726:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158dc00 x1874663333327104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829447.074735:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333327616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.074757:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074761:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.074763:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829447.074766:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333327104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829447.074771:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074777:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074785:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074792:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829447.074792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.074803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829447.074808:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333327744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829447.074813:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333327104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829447.074825:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074829:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.074831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.074836:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074839:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074843:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074848:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829447.074851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829447.074859:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333327872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.074877:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.074880:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.074882:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829447.074888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829447.074891:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11257 00000100:00080000:0.0:1787829447.074897:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.074901:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829447.074908:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829447.074913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158e300 x1874663333327232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829447.074924:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333328000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829447.074937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333327232/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.074950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.074954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.074959:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829447.074966:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829447.074992:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075008:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158f100 x1874663333327360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333327360/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829447.075055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829447.075080:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075092:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158ca80 x1874663333327488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075106:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333327488/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075115:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829447.075129:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829447.075151:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158df80 x1874663333327616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075178:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333327616/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333327616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829447.075204:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333327744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829447.075261:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075276:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158e680 x1874663333327744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075290:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333327744/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075299:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075303:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075307:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333327744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829447.075329:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333327872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829447.075353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075367:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158f800 x1874663333327872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075381:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333327872/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075393:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075397:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333327872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829447.075404:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829447.075426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829447.075438:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9057f158c000 x1874663333328000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829447.075450:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333328000/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829447.075458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829447.075461:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829447.075464:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328000:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829447.458384:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829448.482515:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829449.506438:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829450.530448:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829451.554400:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829452.194318:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194328:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829452.194332:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829452.194342:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194353:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194359:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829452.194400:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333328128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829452.194431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829452.194435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829452.194439:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194456:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829452.194472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333328128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829452.194474:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333328256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829452.194489:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829452.194495:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829452.194500:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194505:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194510:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194516:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829452.194530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333328384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829452.194542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829452.194543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829452.194551:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829452.194556:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194560:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829452.194578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333328512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829452.194585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194587:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829452.194588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829452.194594:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de0000 x1874663333328128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829452.194594:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194598:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194606:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829452.194622:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333328128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829452.194625:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333328640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829452.194649:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829452.194649:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829452.194670:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829452.194675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829452.194678:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333328128:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829452.194690:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194694:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194697:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829452.194710:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333328768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829452.194715:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328256:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829452.194733:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194738:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829452.194740:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829452.194751:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829452.194751:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11262 00000100:00080000:2.0:1787829452.194759:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829452.194766:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829452.194767:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de1c00 x1874663333328256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829452.194771:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829452.194783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333328256/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829452.194789:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333328896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829452.194792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.194799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.194805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829452.194815:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829452.194816:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829452.194823:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829452.194825:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829452.194831:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11262 00000800:00000100:1.0:1787829452.194835:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829452.194841:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829452.194848:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de1500 x1874663333328384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829452.194850:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829452.194862:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829452.194864:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333328384/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.194873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.194876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.194879:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829452.194884:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333329024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829452.194888:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829452.194911:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829452.194920:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de0700 x1874663333328512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829452.194933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333328512/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.194938:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.194940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.194942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829452.194945:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829452.194956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829452.194961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de0a80 x1874663333328640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829452.194968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333328640/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.194973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.194975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.194977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333328640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829452.194988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829452.194999:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829452.195005:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de3800 x1874663333328768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829452.195013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333328768/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.195027:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.195030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.195033:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829452.195040:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829452.195060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829452.195071:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de1880 x1874663333328896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829452.195080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333328896/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.195085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.195088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.195090:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333328896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829452.195094:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829452.195109:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829452.195119:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9057d0de1180 x1874663333329024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829452.195132:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333329024/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829452.195140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829452.195143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829452.195148:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329024:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829452.578372:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829453.602376:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829454.626429:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829455.650460:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829456.674435:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829457.314327:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314336:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829457.314340:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314350:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314360:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314367:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829457.314406:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333329152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829457.314439:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829457.314443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314447:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314452:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829457.314470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333329280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829457.314478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314481:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829457.314482:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314484:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314487:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314489:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314493:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829457.314498:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333329152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829457.314504:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333329408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829457.314513:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314515:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829457.314516:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314519:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314521:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314524:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829457.314535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333329536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829457.314542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314544:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829457.314546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314549:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314552:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314554:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829457.314565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333329664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829457.314572:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829457.314578:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829457.314582:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829457.314583:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314587:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314591:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314594:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314596:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829457.314604:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333329792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829457.314617:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829457.314618:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb221c00 x1874663333329152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829457.314641:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829457.314644:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314653:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314657:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314661:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829457.314666:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333329152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.314682:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829457.314687:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333329920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829457.314690:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.314698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333329152:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829457.314705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829457.314710:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329280:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829457.314710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829457.314713:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829457.314720:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11268 00000100:00080000:3.0:1787829457.314725:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829457.314729:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829457.314734:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829457.314739:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829457.314752:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333330048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829457.314753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb222d80 x1874663333329280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.314787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333329280/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.314794:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.314797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.314800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829457.314832:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829457.314856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.314869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb223100 x1874663333329408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.314883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333329408/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.314897:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.314899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.314902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829457.314906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829457.314923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.314930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb221880 x1874663333329536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.314937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333329536/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.314943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.314945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.314948:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829457.314951:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829457.314961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.314967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb220380 x1874663333329664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.314979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333329664/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.314984:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.314986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.314987:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333329664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829457.314991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829457.315000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.315006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb222300 x1874663333329792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.315012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333329792/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.315017:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.315019:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.315021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829457.315024:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829457.315035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.315040:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb220700 x1874663333329920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.315047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333329920/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.315061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.315064:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.315068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333329920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829457.315074:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829457.315103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829457.315114:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829457/real 1787829457] req@ffff9057fb222a00 x1874663333330048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829457.315122:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333330048/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829457.315128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829457.315130:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829457.315132:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330048:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829457.698380:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829458.722411:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829459.746390:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829460.770380:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829461.794414:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829462.434313:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434325:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829462.434330:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434342:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434354:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434363:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829462.434407:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333330176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829462.434439:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434443:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829462.434445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434451:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434455:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434459:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434471:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829462.434487:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333330304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829462.434497:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333330176:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829462.434502:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829462.434511:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434517:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434522:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434527:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829462.434548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333330432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829462.434559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434563:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829462.434565:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434568:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434572:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829462.434574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829462.434580:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829462.434619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333330560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829462.434630:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829462.434638:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829462.434639:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb221500 x1874663333330176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829462.434648:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434657:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434664:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829462.434674:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333330176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829462.434686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333330688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829462.434690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.434702:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829462.434703:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829462.434710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333330176:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829462.434710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829462.434715:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434723:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434728:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434733:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434739:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829462.434740:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829462.434757:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333330816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829462.434772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829462.434772:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829462.434778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829462.434781:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829462.434786:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434792:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829462.434793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb220e00 x1874663333330304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829462.434800:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434810:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829462.434819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333330304/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829462.434828:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333330944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829462.434832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.434841:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829462.434848:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829462.434850:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330304:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829462.434855:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829462.434859:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829462.434863:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330432:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829462.434867:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11273 00000100:00080000:3.0:1787829462.434873:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829462.434878:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829462.434885:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829462.434896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829462.434901:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333331072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829462.434926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb222680 x1874663333330432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.434943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333330432/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.434952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.434956:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.434960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829462.434967:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829462.434989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829462.435001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb221180 x1874663333330560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.435015:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333330560/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.435023:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.435026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.435029:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829462.435035:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829462.435055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829462.435067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb223b80 x1874663333330688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.435079:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333330688/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.435088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.435091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.435094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333330688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829462.435113:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829462.435133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829462.435145:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb220a80 x1874663333330816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.435157:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333330816/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.435173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.435176:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.435180:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829462.435187:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829462.435218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829462.435264:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb220000 x1874663333330944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.435277:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333330944/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.435285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.435289:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.435292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333330944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829462.435299:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829462.435320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829462.435332:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829462/real 1787829462] req@ffff9057fb223800 x1874663333331072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829462.435344:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333331072/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829462.435352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829462.435355:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829462.435359:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331072:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829462.818440:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829463.842417:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829464.866388:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829465.890464:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829466.914365:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829467.554293:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554302:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829467.554305:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554315:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554325:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554332:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829467.554371:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333331200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829467.554393:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554396:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829467.554397:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554400:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554403:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554405:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829467.554413:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829467.554422:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333331328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829467.554430:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554432:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829467.554434:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554436:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554438:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554440:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829467.554445:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333331200:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829467.554447:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829467.554459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333331456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829467.554466:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554468:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829467.554470:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554472:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554475:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554477:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829467.554481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829467.554489:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333331584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829467.554500:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554503:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829467.554504:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554507:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554509:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829467.554512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829467.554516:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829467.554521:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829467.554530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333331712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829467.554536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829467.554539:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554542:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554544:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554546:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829467.554550:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829467.554574:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333331840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829467.554576:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb223800 x1874663333331200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829467.554601:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554605:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829467.554606:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554611:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00100000:1.0:1787829467.554612:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333331200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829467.554614:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554623:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829467.554629:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829467.554629:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829467.554637:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829467.554641:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333331968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829467.554643:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333331200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829467.554656:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331328:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829467.554657:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829467.554662:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829467.554663:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829467.554667:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11278 00000100:00080000:3.0:1787829467.554670:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829467.554672:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829467.554676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829467.554682:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829467.554686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333332096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829467.554696:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb220000 x1874663333331328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.554706:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333331328/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.554712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.554715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.554717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829467.554736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829467.554754:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.554765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb220a80 x1874663333331456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.554779:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333331456/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.554788:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.554792:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.554795:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829467.554802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829467.554824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.554837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb223b80 x1874663333331584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.554848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333331584/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.554855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.554859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.554863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829467.554869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829467.554892:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.554904:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb221180 x1874663333331712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.554926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333331712/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.554932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.554934:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.554937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333331712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829467.554941:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829467.554959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.554972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb222680 x1874663333331840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.554984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333331840/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.554993:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.554997:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.555001:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829467.555007:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829467.555023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.555031:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb220e00 x1874663333331968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.555044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333331968/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.555055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.555057:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.555059:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333331968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829467.555063:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829467.555074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829467.555080:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829467/real 1787829467] req@ffff9057fb221500 x1874663333332096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829467.555088:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333332096/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829467.555097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829467.555099:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829467.555101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332096:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829467.938405:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829468.962353:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829469.986524:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829471.010400:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829472.034392:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829472.674354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.674369:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829472.674373:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674383:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674395:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674402:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829472.674441:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333332224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.674463:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.674466:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829472.674467:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674471:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674473:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674475:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674483:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829472.674493:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333332352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.674507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829472.674510:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333332224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829472.674511:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.674515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674534:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674540:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829472.674556:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333332480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.674569:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.674572:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829472.674574:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674577:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674580:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674582:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829472.674591:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829472.674598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333332608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.674607:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.674610:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.674611:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674613:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674615:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674618:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829472.674630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333332736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.674636:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829472.674638:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158c000 x1874663333332224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829472.674640:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829472.674643:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674652:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674657:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674660:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829472.674665:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333332224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829472.674675:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333332864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829472.674677:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.674685:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829472.674690:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829472.674692:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333332224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829472.674693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.674697:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674704:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:0.0:1787829472.674708:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.674711:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674716:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829472.674719:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829472.674739:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829472.674744:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333332992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829472.674751:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158f800 x1874663333332352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829472.674766:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829472.674769:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333332352/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829472.674771:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.674775:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.674784:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11283 00000100:00080000:1.0:1787829472.674785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829472.674790:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829472.674791:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.674797:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332352:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829472.674799:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829472.674807:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829472.674807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829472.674825:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333333120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829472.674831:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.674843:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158e680 x1874663333332480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.674852:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333332480/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.674857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.674860:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.674870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829472.674875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829472.674890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.674897:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158df80 x1874663333332608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.674905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333332608/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.674910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.674912:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.674914:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829472.674917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829472.674928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.674934:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158ca80 x1874663333332736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.674941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333332736/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.674946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.674948:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.674949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333332736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829472.674953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829472.674964:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.674974:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158f100 x1874663333332864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.674985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333332864/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.675000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.675004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.675007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829472.675028:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829472.675044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.675051:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158e300 x1874663333332992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.675059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333332992/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.675064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.675066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.675068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333332992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829472.675072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333333120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829472.675082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829472.675088:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff9057f158dc00 x1874663333333120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.675095:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333333120/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.675100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.675102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829472.675104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333333120:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829473.058388:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829474.082460:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829475.106390:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829476.130440:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829477.154428:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829477.794355:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794371:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829477.794375:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794387:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794400:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829477.794408:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829477.794451:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333333248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829477.794471:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794476:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829477.794477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794490:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794494:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829477.794499:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794510:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829477.794528:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333333248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829477.794528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333333376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829477.794545:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794548:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829477.794550:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794554:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794558:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829477.794562:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794567:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829477.794583:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333333504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829477.794590:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794593:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829477.794594:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794597:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794600:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829477.794602:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829477.794605:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794611:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829477.794618:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333333632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829477.794625:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794627:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829477.794629:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794631:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794634:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829477.794636:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794639:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829477.794646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333333760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829477.794652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829477.794657:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb222a00 x1874663333333248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829477.794657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794664:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794668:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829477.794673:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829477.794680:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333333248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829477.794686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333333888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829477.794690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.794697:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.794702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333333248:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829477.794705:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794710:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829477.794713:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829477.794718:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00080000:3.0:1787829477.794722:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829477.794725:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333376:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829477.794729:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794736:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829477.794745:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829477.794751:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333334016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829477.794756:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb220700 x1874663333333376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.794776:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333333376/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829477.794778:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829477.794786:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829477.794788:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829477.794789:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829477.794793:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829477.794797:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11288 00000100:00100000:1.0:1787829477.794798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333376:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829477.794802:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829477.794806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333504:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829477.794809:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829477.794817:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829477.794830:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333334144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829477.794838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.794853:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb222300 x1874663333333504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.794866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333333504/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.794873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.794877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.794880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829477.794886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829477.794905:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.794916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb220380 x1874663333333632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.794927:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333333632/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.794934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.794937:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.794941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829477.794946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829477.794966:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.794976:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb221880 x1874663333333760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.794994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333333760/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.795002:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.795004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.795008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333333760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829477.795013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333333888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829477.795032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.795044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb223100 x1874663333333888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.795055:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333333888/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.795063:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.795066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.795069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333333888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829477.795088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829477.795106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.795117:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb222d80 x1874663333334016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.795128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333334016/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.795141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.795145:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.795148:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829477.795155:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829477.795173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829477.795183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff9057fb221c00 x1874663333334144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.795203:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333334144/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.795211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.795215:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829477.795218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334144:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829478.178556:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829479.202401:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829480.226380:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829481.250401:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829482.274370:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829482.914330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914341:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829482.914345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914355:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914367:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829482.914375:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829482.914414:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333334272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829482.914440:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914444:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829482.914458:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914467:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829482.914472:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829482.914482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829482.914490:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333334272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829482.914498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333334400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829482.914511:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914514:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829482.914516:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914521:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914525:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829482.914529:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829482.914534:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829482.914546:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333334528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829482.914553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829482.914561:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914565:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829482.914568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914573:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914577:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829482.914581:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829482.914587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829482.914596:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158d880 x1874663333334272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829482.914600:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333334656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829482.914616:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829482.914619:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333334272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829482.914621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829482.914624:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914632:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914635:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829482.914639:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829482.914640:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829482.914645:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:0.0:1787829482.914647:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829482.914650:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333334272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829482.914660:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333334784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829482.914668:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914672:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829482.914674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334400:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829482.914675:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914682:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914687:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829482.914691:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829482.914697:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829482.914697:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829482.914712:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333334912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829482.914714:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158ce00 x1874663333334400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829482.914739:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829482.914743:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333334400/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829482.914745:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829482.914749:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914757:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:1.0:1787829482.914758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829482.914762:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829482.914765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829482.914770:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829482.914773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334400:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787829482.914777:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829482.914782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829482.914791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333335040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829482.914801:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829482.914807:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829482.914807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829482.914811:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829482.914816:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11293 00000100:00080000:0.0:1787829482.914820:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829482.914823:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158c700 x1874663333334528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829482.914827:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829482.914836:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829482.914842:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333334528/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.914853:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829482.914858:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333335168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829482.914858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.914863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829482.914875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829482.914900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829482.914911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158ea00 x1874663333334656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829482.914923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333334656/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.914931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829482.914933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.914937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829482.914943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829482.914960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829482.914969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158d180 x1874663333334784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829482.914980:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333334784/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.914985:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829482.914987:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.914989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333334784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829482.915001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829482.915014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829482.915020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158d500 x1874663333334912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829482.915026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333334912/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.915035:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829482.915036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.915038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333334912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829482.915041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829482.915050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829482.915054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158ed80 x1874663333335040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829482.915060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333335040/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.915065:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829482.915066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.915068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829482.915070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829482.915079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829482.915083:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829482] req@ffff9057f158fb80 x1874663333335168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829482.915089:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333335168/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829482.915093:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829482.915095:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829482.915097:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335168:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829483.298457:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829484.322514:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829485.346408:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829486.370399:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829487.394490:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829488.034371:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034383:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829488.034388:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829488.034399:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034423:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034431:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829488.034473:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333335296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829488.034503:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034507:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829488.034508:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829488.034513:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034516:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034521:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829488.034543:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333335424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829488.034555:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829488.034561:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333335296:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829488.034561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829488.034566:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829488.034573:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034577:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034581:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034586:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829488.034602:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333335552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829488.034613:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034616:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829488.034619:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829488.034623:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034628:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034633:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829488.034638:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034643:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829488.034660:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333335680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829488.034672:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034676:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829488.034678:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829488.034682:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb221c00 x1874663333335296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829488.034687:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034697:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034704:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829488.034707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333335296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829488.034711:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829488.034717:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.034722:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.034727:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333335296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829488.034731:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333335808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829488.034744:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034748:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829488.034750:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829488.034759:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335424:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829488.034759:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034765:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034772:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034778:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829488.034792:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829488.034796:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333335936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829488.034806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb222d80 x1874663333335424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.034836:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333335424/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829488.034838:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829488.034849:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829488.034852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829488.034853:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829488.034857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.034864:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335424:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829488.034865:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00100000:1.0:1787829488.034872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829488.034874:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829488.034884:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034891:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829488.034896:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829488.034913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb223100 x1874663333335552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829488.034913:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333336064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829488.034944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333335552/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829488.034949:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829488.034952:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.034958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829488.034959:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829488.034962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829488.034964:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829488.034969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335680:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829488.034974:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11298 00000100:00080000:3.0:1787829488.034980:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829488.034987:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829488.034988:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829488.034995:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829488.034998:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb221880 x1874663333335680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.035007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333335680/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.035013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.035014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.035016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829488.035023:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829488.035028:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333336192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829488.035037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829488.035047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb220380 x1874663333335808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.035061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333335808/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.035067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.035069:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.035071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333335808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829488.035085:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829488.035109:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829488.035120:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb222300 x1874663333335936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.035128:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333335936/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.035133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.035136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.035138:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333335936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.035142:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829488.035154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829488.035164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb220700 x1874663333336064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.035175:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333336064/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.035191:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.035195:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.035199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.035205:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829488.035250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829488.035260:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9057fb222a00 x1874663333336192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.035268:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333336192/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.035273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.035276:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829488.035278:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336192:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829488.418445:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829489.442451:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829490.466390:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829491.490482:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829492.514400:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829493.154297:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154308:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829493.154312:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154324:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154337:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154345:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829493.154404:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333336320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829493.154433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154437:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829493.154439:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154444:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154454:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829493.154471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333336320:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829493.154474:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829493.154492:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333336448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829493.154505:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829493.154510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154515:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154519:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154523:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829493.154529:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829493.154540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829493.154542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333336576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829493.154555:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829493.154562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154566:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154570:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154575:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829493.154581:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829493.154585:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb221500 x1874663333336320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829493.154597:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333336704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829493.154604:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333336320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829493.154615:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829493.154620:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829493.154620:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829493.154624:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829493.154627:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829493.154632:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00100000:1.0:1787829493.154633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333336320:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829493.154637:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154643:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829493.154648:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829493.154660:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333336832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829493.154664:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336448:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829493.154678:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154683:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829493.154685:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154690:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154695:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154700:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829493.154704:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829493.154709:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829493.154720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb220e00 x1874663333336448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829493.154724:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333336960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829493.154747:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333336448/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829493.154748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829493.154756:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829493.154759:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154768:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:1.0:1787829493.154770:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829493.154773:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829493.154780:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829493.154782:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829493.154790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336448:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787829493.154790:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829493.154803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829493.154809:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333337088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829493.154822:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829493.154826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829493.154828:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829493.154831:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829493.154836:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11303 00000100:00080000:3.0:1787829493.154839:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829493.154845:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829493.154846:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb222680 x1874663333336576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829493.154849:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829493.154861:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333337216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829493.154870:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333336576/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.154882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.154887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.154890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.154895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.154920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829493.154929:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb221180 x1874663333336704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.154937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333336704/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.154943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.154945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.154947:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829493.154959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.154970:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829493.154977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb223b80 x1874663333336832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.154984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333336832/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.154989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.154991:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.154993:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333336832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829493.154996:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829493.155007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829493.155012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb220a80 x1874663333336960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.155020:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333336960/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.155029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.155032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.155034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333336960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.155037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829493.155046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829493.155052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb220000 x1874663333337088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.155059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333337088/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.155064:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.155066:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.155068:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.155071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.155081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829493.155086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9057fb223800 x1874663333337216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.155092:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333337216/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.155097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.155099:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829493.155102:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337216:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829493.538414:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829494.562386:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829495.586386:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829496.610388:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829497.634410:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829498.274319:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274329:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829498.274334:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274356:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274368:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274376:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829498.274420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333337344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829498.274450:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274454:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829498.274456:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274470:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274481:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829498.274497:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333337472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829498.274504:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333337344:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829498.274518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274521:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829498.274522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274526:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274530:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274533:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274538:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829498.274549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333337600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829498.274556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829498.274560:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274563:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274565:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274570:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829498.274570:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829498.274585:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333337728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829498.274592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274594:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829498.274595:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274599:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274602:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274605:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274608:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829498.274616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333337856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829498.274620:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158fb80 x1874663333337344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829498.274628:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829498.274635:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829498.274638:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274645:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274650:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274654:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829498.274658:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333337344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829498.274664:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333337984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829498.274683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829498.274684:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829498.274687:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829498.274691:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829498.274691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829498.274699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00100000:1.0:1787829498.274700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333337344:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829498.274704:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274709:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829498.274719:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333338112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829498.274728:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829498.274728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337472:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829498.274737:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829498.274739:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829498.274742:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11309 00000100:00080000:0.0:1787829498.274745:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829498.274747:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829498.274750:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829498.274757:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333338240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829498.274773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.274787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158ed80 x1874663333337472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.274797:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333337472/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.274802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.274804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.274807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829498.274811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829498.274825:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.274832:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158d500 x1874663333337600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.274840:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333337600/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.274845:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.274846:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.274848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829498.274852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829498.274863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.274869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158d180 x1874663333337728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.274876:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333337728/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.274882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.274885:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.274889:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829498.274895:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829498.274914:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.274926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158ea00 x1874663333337856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.274945:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333337856/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.274951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.274953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.274955:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333337856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829498.274960:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829498.274974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.274981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158c700 x1874663333337984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.274989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333337984/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.274994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.274996:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.274998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333337984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829498.275012:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333338112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829498.275023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.275029:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158ce00 x1874663333338112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.275037:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333338112/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.275046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.275049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.275051:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333338112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829498.275054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333338240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829498.275064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829498.275074:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829498/real 1787829498] req@ffff9057f158d880 x1874663333338240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829498.275086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333338240/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829498.275094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829498.275098:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829498.275101:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333338240:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829498.658494:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829499.682391:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829500.706406:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829501.730379:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829502.754435:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829503.394347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394355:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829503.394359:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394369:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394379:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394386:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829503.394428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333338368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829503.394468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394473:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394478:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829503.394505:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333338496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394517:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829503.394521:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333338368:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829503.394527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394533:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394535:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394538:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829503.394552:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333338624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394562:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829503.394564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394572:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829503.394583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333338752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394590:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829503.394598:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829503.394599:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829503.394603:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394607:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394609:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829503.394621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333338880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394631:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829503.394632:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394635:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394637:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829503.394640:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829503.394650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333339008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394656:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829503.394661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829503.394664:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394666:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829503.394670:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78abb80 x1874663333338368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829503.394671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829503.394676:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829503.394685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333339136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829503.394694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829503.394697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829503.394698:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829503.394703:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333338368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829503.394712:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11314 00000100:00080000:1.0:1787829503.394719:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829503.394720:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829503.394725:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829503.394731:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829503.394732:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829503.394740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333338368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829503.394748:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333339264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829503.394771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829503.394800:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.394824:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78aa680 x1874663333338496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.394837:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333338496/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.394847:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.394850:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.394854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.394861:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829503.394880:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.394892:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78a9c00 x1874663333338624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.394905:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333338624/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.394920:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.394924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.394927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.394943:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829503.394967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.394979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78aa300 x1874663333338752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.394991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333338752/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.394999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.395003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.395006:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829503.395013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829503.395044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.395056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78a9180 x1874663333338880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.395068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333338880/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.395077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.395080:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.395085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333338880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829503.395100:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829503.395120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.395132:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78a8700 x1874663333339008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.395145:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333339008/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.395153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.395156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.395160:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.395166:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829503.395192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.395206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78a9500 x1874663333339136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.395218:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333339136/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.395255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.395259:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.395262:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.395269:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829503.395289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829503.395301:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829503/real 1787829503] req@ffff9057c78ab100 x1874663333339264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.395332:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333339264/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.395340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.395343:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829503.395346:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333339264:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829503.778376:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829504.802424:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829505.826615:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829506.850430:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829507.874412:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829508.514311:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514320:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829508.514324:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514334:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514343:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829508.514350:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829508.514385:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333339392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514410:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514413:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829508.514414:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514418:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514420:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829508.514422:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514431:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829508.514440:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333339520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829508.514454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514456:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829508.514463:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333339392:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829508.514464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829508.514478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333339648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829508.514489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514491:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514494:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829508.514497:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829508.514508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333339776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829508.514519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514521:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514523:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829508.514525:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829508.514535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333339904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514542:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514544:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829508.514546:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514549:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514551:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829508.514554:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514557:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829508.514564:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333340032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829508.514574:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339520:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829508.514582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514588:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829508.514590:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514596:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514601:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829508.514605:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339648:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829508.514610:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514619:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829508.514628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829508.514647:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333340160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829508.514656:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829508.514659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829508.514661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829508.514664:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11319 00000100:00080000:1.0:1787829508.514667:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829508.514672:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339904:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829508.514673:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829508.514681:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829508.514693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333340288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829508.514712:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829508.514740:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829508.514753:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340288:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829508.898464:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829509.922355:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829510.946464:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829511.586517:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829511.586562:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829511.586582:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586626:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586637:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586645:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586654:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586665:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586676:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829511.586687:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829511.586699:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78ab100 x1874663333339520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333339520/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586744:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586759:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829511.586768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78a9500 x1874663333339648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333339648/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586799:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829511.586806:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78a8700 x1874663333339776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586819:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333339776/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586831:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829511.586841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78a9180 x1874663333339904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586854:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333339904/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586863:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333339904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829511.586877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78a8a80 x1874663333339392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586891:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333339392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586907:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333339392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829511.586937:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78aa300 x1874663333340032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.586950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333340032/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.586971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.586976:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.586980:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829511.586988:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78a9c00 x1874663333340160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.587000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333340160/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.587009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.587013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.587016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829511.587023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829508/real 1787829511] req@ffff9057c78aa680 x1874663333340288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829511.587036:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333340288/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829511.587044:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829511.587048:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829511.587052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333340288:192.168.203.156@tcp:8:kworker.0 00010000:00010000:2.0:1787829511.970413:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829512.994500:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829513.634333:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829513.634346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634355:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634365:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634371:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829513.634406:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333340416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829513.634432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634435:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829513.634437:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634443:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829513.634455:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829513.634464:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333340544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829513.634473:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829513.634476:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333340416:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829513.634477:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829513.634487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634493:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634498:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634503:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829513.634509:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829513.634524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333340672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829513.634535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829513.634541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634545:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634549:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634556:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829513.634559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829513.634563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829513.634579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333340800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829513.634591:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634595:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829513.634597:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634601:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634605:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829513.634614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de1f80 x1874663333340416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.634614:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829513.634623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829513.634644:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333340928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829513.634653:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333340416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.634657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634668:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829513.634672:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.634679:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.634679:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634688:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.634689:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829513.634699:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829513.634700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333340416:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787829513.634709:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829513.634725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333341056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829513.634736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340544:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829513.634739:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829513.634746:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829513.634748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634761:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:2.0:1787829513.634767:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634772:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829513.634778:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829513.634789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829513.634795:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333341184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829513.634808:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829513.634809:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de3100 x1874663333340544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.634823:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829513.634827:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333340544/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.634828:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829513.634839:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11324 00000100:00080000:1.0:1787829513.634849:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.634849:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829513.634858:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829513.634859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.634869:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340544:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1787829513.634870:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829513.634889:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333341312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829513.634894:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829513.634927:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.634941:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de2a00 x1874663333340672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.634955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333340672/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.634970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.634974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.634977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829513.634983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829513.635005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.635017:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de0e00 x1874663333340800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.635029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333340800/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.635038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.635041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.635045:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829513.635051:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829513.635080:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.635093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de1180 x1874663333340928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.635103:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333340928/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.635111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.635114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.635117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333340928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829513.635135:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829513.635157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.635171:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de1880 x1874663333341056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.635186:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333341056/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.635195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.635198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.635203:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829513.635212:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829513.635442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.635456:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de3800 x1874663333341184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.635471:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333341184/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.635480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.635484:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.635487:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829513.635494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829513.635514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829513.635527:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff9057d0de0a80 x1874663333341312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829513.635538:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333341312/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829513.635547:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829513.635551:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829513.635554:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333341312:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829514.018399:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829515.042440:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829516.066335:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829517.090401:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829518.114380:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829518.754354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754363:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829518.754367:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754376:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754386:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754392:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829518.754437:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333341440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754461:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754464:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829518.754466:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754469:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754472:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754474:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829518.754482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829518.754497:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333341568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754505:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829518.754509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754512:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754515:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754521:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754522:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333341440:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1787829518.754525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829518.754535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333341696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754542:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754545:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829518.754546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754549:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754551:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754553:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829518.754557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829518.754565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333341824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829518.754575:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754577:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754580:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754582:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829518.754585:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829518.754589:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829518.754599:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333341952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754612:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754614:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829518.754616:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754618:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754621:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829518.754626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829518.754637:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158dc00 x1874663333341440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829518.754639:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333342080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829518.754672:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754675:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829518.754676:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829518.754681:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333341440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829518.754684:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754690:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829518.754697:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829518.754701:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829518.754702:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829518.754710:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829518.754715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333342208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829518.754716:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333341440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829518.754729:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341568:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829518.754731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829518.754735:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829518.754736:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829518.754745:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11329 00000100:00080000:0.0:1787829518.754750:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829518.754752:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829518.754756:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829518.754761:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829518.754765:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158e300 x1874663333341568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829518.754773:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333342336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829518.754781:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333341568/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.754791:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.754793:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.754796:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829518.754822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829518.754836:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754843:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158f100 x1874663333341696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.754851:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333341696/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.754857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.754859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.754861:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829518.754864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829518.754875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754880:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158ca80 x1874663333341824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.754888:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333341824/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.754893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.754894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.754897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829518.754901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829518.754911:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158df80 x1874663333341952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.754923:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333341952/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.754929:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.754930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.754932:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333341952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829518.754941:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829518.754953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158e680 x1874663333342080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.754967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333342080/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.754972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.754973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.754975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829518.754979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829518.754989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.754995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158f800 x1874663333342208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.755007:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333342208/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.755012:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.755014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.755016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829518.755024:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829518.755037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829518.755043:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff9057f158c000 x1874663333342336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829518.755050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333342336/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829518.755055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829518.755058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829518.755060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333342336:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829519.138447:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829520.162491:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829521.186404:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829522.210441:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829523.234417:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829523.874330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874338:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829523.874342:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874351:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874361:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874367:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829523.874404:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333342464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829523.874428:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874431:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829523.874432:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874435:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874438:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874440:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874448:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829523.874459:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333342592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829523.874469:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874472:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829523.874472:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333342464:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829523.874476:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874483:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874486:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874488:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874491:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829523.874500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333342720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829523.874506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829523.874510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874512:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874526:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874529:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829523.874539:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829523.874541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333342848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829523.874549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874551:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829523.874553:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874556:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874559:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874562:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874564:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829523.874572:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333342976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829523.874578:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829523.874580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158c000 x1874663333342464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829523.874581:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829523.874584:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874591:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874595:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874600:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829523.874614:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333343104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829523.874619:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333342464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829523.874622:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874628:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829523.874631:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874637:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874642:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874649:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829523.874650:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874657:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829523.874658:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333343232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829523.874664:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333342464:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829523.874679:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829523.874681:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829523.874682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829523.874685:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11334 00000100:00080000:0.0:1787829523.874687:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829523.874690:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829523.874692:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829523.874698:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333343360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829523.874703:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829523.874733:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874745:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158f800 x1874663333342592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874755:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333342592/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874766:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829523.874770:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829523.874781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158e680 x1874663333342720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333342720/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874800:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874804:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829523.874808:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829523.874819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874825:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158df80 x1874663333342848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333342848/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874839:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874841:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829523.874844:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829523.874854:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158ca80 x1874663333342976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874867:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333342976/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333342976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829523.874878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829523.874888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158f100 x1874663333343104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333343104/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874906:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829523.874913:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829523.874923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.874930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158e300 x1874663333343232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.874941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333343232/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.874949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.874953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.874957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829523.874978:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829523.874999:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829523.875009:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff9057f158dc00 x1874663333343360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829523.875021:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333343360/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829523.875036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829523.875039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829523.875042:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333343360:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829524.258410:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829525.282384:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829526.306379:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829527.330520:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829528.354402:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829528.994343:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994352:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829528.994356:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994365:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994374:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829528.994418:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333343488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994444:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994446:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829528.994448:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994451:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994454:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994457:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994465:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829528.994476:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333343616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829528.994488:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994490:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994492:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994494:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994497:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829528.994501:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333343488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829528.994507:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333343744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994516:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994518:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829528.994520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994522:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994524:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994526:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994530:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829528.994539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333343872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829528.994554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994557:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994559:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994561:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994565:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829528.994569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829528.994575:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333344000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994582:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829528.994587:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:0.0:1787829528.994592:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994595:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829528.994606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333344128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829528.994611:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994613:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829528.994614:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829528.994620:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00100000:1.0:1787829528.994621:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158d880 x1874663333343488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829528.994623:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829528.994628:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829528.994635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829528.994644:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333344256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829528.994650:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333343488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829528.994655:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829528.994663:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829528.994665:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829528.994667:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829528.994669:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11339 00000100:00080000:1.0:1787829528.994674:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:0.0:1787829528.994675:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829528.994680:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333343488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829528.994685:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994690:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343616:192.168.203.156@tcp:38:kworker.0 00000008:00100000:0.0:1787829528.994694:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829528.994712:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333344384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829528.994715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994728:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158ce00 x1874663333343616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994740:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333343616/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994746:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994749:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994752:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829528.994772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829528.994784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994790:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158c700 x1874663333343744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994797:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333343744/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994806:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829528.994809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829528.994819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994825:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158ea00 x1874663333343872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994832:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333343872/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994838:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333343872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829528.994843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829528.994853:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994859:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158d180 x1874663333344000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333344000/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829528.994878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829528.994888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158d500 x1874663333344128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333344128/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994907:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829528.994912:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829528.994921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158ed80 x1874663333344256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333344256/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829528.994946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829528.994955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829528.994961:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff9057f158fb80 x1874663333344384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829528.994967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333344384/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829528.994972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829528.994973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829528.994975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333344384:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829529.378410:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829530.402453:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829531.426403:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829532.450426:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829533.474422:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829534.114332:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114340:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829534.114349:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114372:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114380:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829534.114428:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158fb80 x1874663333344512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114453:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114456:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829534.114457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114468:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114477:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829534.114486:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ed80 x1874663333344640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114495:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829534.114496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333344512:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829534.114498:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829534.114502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114507:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114512:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114521:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829534.114531:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d500 x1874663333344768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114539:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114542:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829534.114543:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114545:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114549:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114552:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114556:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829534.114562:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829534.114566:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d180 x1874663333344896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114577:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829534.114578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114581:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114584:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114586:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114588:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829534.114597:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ea00 x1874663333345024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829534.114605:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158fb80 x1874663333344512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829534.114609:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829534.114617:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114623:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114626:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829534.114628:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158fb80 x1874663333344512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829534.114632:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114639:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829534.114650:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829534.114652:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c700 x1874663333345152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829534.114661:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.114672:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333344512:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829534.114673:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829534.114684:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114688:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114691:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829534.114693:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114695:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829534.114701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ce00 x1874663333345280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829534.114707:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829534.114709:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829534.114710:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829534.114713:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11344 00000100:00080000:0.0:1787829534.114716:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829534.114719:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344640:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829534.114727:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829534.114732:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829534.114742:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158d880 x1874663333345408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829534.114751:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.114767:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158ed80 x1874663333344640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.114783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ed80 x1874663333344640/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.114793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.114798:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.114802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829534.114809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829534.114829:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.114837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158d500 x1874663333344768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.114845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d500 x1874663333344768/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.114851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.114853:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.114856:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829534.114859:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829534.114871:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.114878:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158d180 x1874663333344896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.114886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d180 x1874663333344896/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.114899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.114902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.114904:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333344896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829534.114908:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829534.114920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.114927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158ea00 x1874663333345024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.114937:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ea00 x1874663333345024/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.114946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.114949:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.114953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829534.114959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829534.114980:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.114991:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158c700 x1874663333345152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.114999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c700 x1874663333345152/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.115009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.115012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.115014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829534.115018:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829534.115031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.115037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158ce00 x1874663333345280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.115045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ce00 x1874663333345280/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.115050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.115052:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.115053:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829534.115068:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829534.115079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829534.115086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9057f158d880 x1874663333345408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829534.115111:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158d880 x1874663333345408/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829534.115117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829534.115119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829534.115121:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333345408:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829534.498495:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829535.522403:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829536.546407:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829537.570412:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829538.594387:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829539.234345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234357:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829539.234360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829539.234371:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00080000:2.0:1787829539.234384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829539.234392:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829539.234433:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333345536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829539.234464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829539.234471:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829539.234475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00080000:2.0:1787829539.234479:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829539.234484:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829539.234495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829539.234522:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333345664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829539.234536:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333345536:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829539.234539:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234552:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829539.234554:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829539.234560:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00080000:2.0:1787829539.234564:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829539.234570:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829539.234576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829539.234591:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333345792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829539.234604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829539.234611:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829539.234618:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11349 00000800:00000100:1.0:1787829539.234623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829539.234628:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829539.234636:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829539.234642:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829539.234660:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333345920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829539.234671:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829539.234678:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de0700 x1874663333345536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234678:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829539.234686:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829539.234702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00080000:2.0:1787829539.234709:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829539.234713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333345536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234718:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829539.234735:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829539.234738:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.234745:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.234750:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333345536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829539.234758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333346048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829539.234772:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829539.234793:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829539.234800:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234806:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829539.234809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829539.234811:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de1500 x1874663333345664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234821:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00100000:1.0:1787829539.234831:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333345664/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234833:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829539.234846:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829539.234847:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829539.234857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.234861:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345664:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1787829539.234861:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829539.234873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829539.234885:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333346176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829539.234893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829539.234910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de1c00 x1874663333345792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234911:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829539.234928:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829539.234935:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829539.234936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333345792/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.234950:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00080000:1.0:1787829539.234951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.234963:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829539.234963:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829539.234973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345792:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829539.234979:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829539.234986:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345920:192.168.203.156@tcp:8:kworker.0 00000008:00100000:2.0:1787829539.234991:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829539.235004:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829539.235013:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333346304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829539.235015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de0000 x1874663333345920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829539.235044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333345920/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829539.235087:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829539.235088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829539.235099:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829539.235103:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829539.235104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829539.235118:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11349 00000100:00100000:1.0:1787829539.235119:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333345920:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787829539.235129:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829539.235137:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829539.235144:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829539.235155:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829539.235163:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333346432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829539.235188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829539.235209:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de0a80 x1874663333346048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829539.235269:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333346048/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829539.235278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.235282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.235285:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829539.235292:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829539.235311:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829539.235322:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de3800 x1874663333346176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829539.235332:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333346176/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829539.235340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.235343:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.235346:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829539.235352:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829539.235370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829539.235381:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de1880 x1874663333346304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829539.235393:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333346304/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829539.235403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.235406:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.235408:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829539.235419:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829539.235434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829539.235441:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829539/real 1787829539] req@ffff9057d0de1180 x1874663333346432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829539.235448:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333346432/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829539.235453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829539.235455:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829539.235457:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333346432:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829539.618352:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829540.642363:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829541.666500:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829542.690344:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829543.714383:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829544.354319:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354332:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829544.354336:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354360:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354368:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829544.354406:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333346560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829544.354432:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354434:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829544.354436:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354438:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354441:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354443:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354451:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829544.354460:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333346688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829544.354479:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354484:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829544.354486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333346560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829544.354487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354501:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354505:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354511:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829544.354526:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333346816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829544.354534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829544.354537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354540:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354543:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354546:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829544.354558:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333346944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829544.354561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829544.354570:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354573:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829544.354574:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354578:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354580:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354583:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354586:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829544.354594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333347072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829544.354600:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829544.354603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354607:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354609:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354612:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829544.354621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333347200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829544.354626:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de0e00 x1874663333346560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829544.354630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829544.354642:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354646:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354649:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829544.354654:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333346560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829544.354654:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829544.354670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829544.354673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333347328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829544.354677:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354685:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333346560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829544.354688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829544.354694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829544.354695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829544.354699:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11355 00000100:00080000:2.0:1787829544.354701:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829544.354703:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829544.354706:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829544.354713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829544.354715:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333347456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829544.354737:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354750:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de2a00 x1874663333346688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354759:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333346688/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354764:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354767:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354769:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829544.354773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829544.354783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de3100 x1874663333346816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354797:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333346816/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354805:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829544.354808:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829544.354824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de1f80 x1874663333346944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354847:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333346944/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354856:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333346944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829544.354878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829544.354894:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de1180 x1874663333347072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333347072/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829544.354923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829544.354934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de1880 x1874663333347200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354947:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333347200/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354957:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829544.354964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829544.354974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.354979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de3800 x1874663333347328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.354986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333347328/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.354990:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.354993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.354995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829544.355008:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829544.355035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829544.355044:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829544/real 1787829544] req@ffff9057d0de0a80 x1874663333347456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829544.355052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333347456/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829544.355058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829544.355060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829544.355063:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333347456:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829544.738364:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829545.762386:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829546.786396:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829547.810442:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829548.834477:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829549.474330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829549.474348:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829549.474360:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829549.474381:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829549.474422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333347584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829549.474449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474452:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829549.474454:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829549.474457:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829549.474462:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474470:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829549.474480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333347712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829549.474492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333347584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829549.474494:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829549.474504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829549.474509:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829549.474520:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829549.474540:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333347840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829549.474548:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474551:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829549.474552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829549.474555:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474559:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829549.474561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829549.474570:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829549.474580:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333347968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829549.474589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474591:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829549.474593:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829549.474595:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829549.474599:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474603:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829549.474614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de0000 x1874663333347584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829549.474623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333348096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829549.474633:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829549.474635:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333347584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829549.474644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829549.474649:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829549.474650:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829549.474657:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:1.0:1787829549.474664:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829549.474666:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829549.474670:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333347584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829549.474676:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474683:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829549.474696:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829549.474697:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333348224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829549.474715:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474720:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829549.474723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829549.474727:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829549.474732:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474739:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829549.474742:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de1c00 x1874663333347712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829549.474748:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474757:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829549.474762:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333347712/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.474769:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.474771:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.474773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829549.474780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333348352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829549.474781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347840:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829549.474801:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829549.474805:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829549.474807:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829549.474811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829549.474815:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11360 00000100:00080000:2.0:1787829549.474820:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829549.474824:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de1500 x1874663333347840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829549.474827:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829549.474837:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829549.474842:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333347840/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829549.474849:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333348480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829549.474851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.474859:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.474866:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829549.474873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829549.474888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829549.474895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de0700 x1874663333347968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829549.474903:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333347968/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.474908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.474910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.474911:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333347968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829549.474915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829549.474925:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829549.474931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de0a80 x1874663333348096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829549.474944:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333348096/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.474949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.474950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.474952:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829549.474963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829549.474974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829549.474980:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de3800 x1874663333348224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829549.475003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333348224/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.475008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.475010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.475013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829549.475016:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829549.475029:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829549.475035:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de1880 x1874663333348352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829549.475042:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333348352/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.475051:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.475053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.475054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829549.475058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829549.475068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829549.475073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829549/real 1787829549] req@ffff9057d0de1180 x1874663333348480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829549.475080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333348480/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829549.475085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829549.475087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829549.475089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333348480:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829549.858430:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829550.882379:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829551.906437:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829552.930404:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829553.954461:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829554.594345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594355:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829554.594360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594369:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00080000:1.0:1787829554.594380:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594387:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829554.594431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333348608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829554.594460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594463:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829554.594464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594468:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00080000:1.0:1787829554.594472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829554.594497:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333348608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829554.594509:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333348736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829554.594523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829554.594529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594534:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00080000:1.0:1787829554.594539:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594543:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829554.594565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829554.594567:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333348864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829554.594581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829554.594587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594591:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00080000:1.0:1787829554.594596:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594601:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829554.594608:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78abb80 x1874663333348608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829554.594609:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829554.594628:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333348992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829554.594630:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333348608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.594655:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.594662:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829554.594666:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594669:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829554.594670:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594674:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00100000:2.0:1787829554.594677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333348608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829554.594679:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594686:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829554.594699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333349120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829554.594705:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829554.594715:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829554.594720:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829554.594723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594730:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11365 00000800:00000100:2.0:1787829554.594731:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829554.594739:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829554.594745:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78aad80 x1874663333348736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.594748:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829554.594763:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333348736/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.594771:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.594774:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829554.594780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333349248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829554.594780:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829554.594789:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829554.594805:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829554.594811:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829554.594812:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829554.594816:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829554.594823:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00100000:2.0:1787829554.594825:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78a8380 x1874663333348864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.594829:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594837:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594847:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829554.594852:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333348864/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829554.594864:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333349376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829554.594866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.594877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829554.594882:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829554.594885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829554.594888:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829554.594895:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829554.594900:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348992:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829554.594903:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11365 00000100:00080000:1.0:1787829554.594916:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829554.594922:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829554.594929:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829554.594934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829554.594960:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333349504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829554.594972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78aaa00 x1874663333348992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829554.594989:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333348992/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.594999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.595003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829554.595007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333348992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829554.595025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829554.595054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829554.595068:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78ab480 x1874663333349120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829554.595081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333349120/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.595090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.595094:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829554.595098:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829554.595104:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829554.595131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829554.595144:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78ab800 x1874663333349248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829554.595157:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333349248/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.595166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.595170:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829554.595174:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829554.595180:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829554.595201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829554.595212:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78a9880 x1874663333349376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829554.595248:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333349376/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.595258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.595262:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829554.595265:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829554.595272:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829554.595294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829554.595306:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff9057c78a8000 x1874663333349504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829554.595319:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333349504/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829554.595328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829554.595332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829554.595336:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333349504:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829554.978379:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829556.002478:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829557.026347:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829558.050427:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829559.074437:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829559.714317:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714329:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829559.714333:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714342:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714353:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714360:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829559.714397:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333349632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829559.714421:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714424:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829559.714426:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714429:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714432:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714435:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714444:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829559.714454:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333349760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829559.714459:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333349632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829559.714471:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714474:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829559.714475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714480:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714482:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714484:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714488:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829559.714498:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333349888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829559.714505:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829559.714509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714512:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714520:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714522:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829559.714532:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829559.714537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333350016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829559.714545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829559.714549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714552:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714554:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714557:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714560:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829559.714570:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333350144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829559.714577:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de1f80 x1874663333349632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829559.714581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829559.714585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714590:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714600:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829559.714610:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333349632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829559.714615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333350272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829559.714641:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714645:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829559.714647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829559.714648:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829559.714653:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714655:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714658:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829559.714661:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:2.0:1787829559.714663:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829559.714667:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333349632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829559.714673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333350400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829559.714681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829559.714684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829559.714685:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829559.714690:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349760:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829559.714690:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11370 00000100:00080000:2.0:1787829559.714695:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829559.714700:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829559.714704:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829559.714714:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333350528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829559.714721:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de3100 x1874663333349760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714748:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333349760/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714757:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.714759:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.714763:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829559.714775:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714781:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de2a00 x1874663333349888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714789:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333349888/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714796:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.714798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333349888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.714802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829559.714814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714820:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de0e00 x1874663333350016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333350016/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.714836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829559.714840:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829559.714850:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714856:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de1180 x1874663333350144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333350144/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714869:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.714871:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829559.714875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829559.714885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de1880 x1874663333350272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333350272/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714906:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.714929:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.714936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829559.714960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.714972:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de3800 x1874663333350400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.714985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333350400/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.714995:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.714999:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.715003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.715026:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829559.715049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829559.715064:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff9057d0de0a80 x1874663333350528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.715077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333350528/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.715093:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.715096:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829559.715100:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333350528:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829560.098440:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829561.122425:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829562.146408:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829563.170467:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829564.194373:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829564.834338:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834347:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829564.834351:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834360:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834371:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834378:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829564.834416:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333350656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829564.834440:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829564.834444:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834447:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834452:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834460:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829564.834468:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333350784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829564.834481:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829564.834495:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333350656:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829564.834496:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834503:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834506:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834508:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834512:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829564.834521:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333350912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829564.834529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834531:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829564.834533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834535:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834546:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834549:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829564.834558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829564.834567:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333351040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829564.834575:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834578:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829564.834579:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834582:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834585:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834588:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834591:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829564.834598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333351168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829564.834604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834608:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829564.834610:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829564.834610:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158dc00 x1874663333350656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829564.834617:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834622:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834628:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834632:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829564.834641:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333351296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829564.834643:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333350656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829564.834656:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829564.834663:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829564.834666:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834670:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834673:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829564.834674:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829564.834676:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834682:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829564.834684:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1787829564.834690:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333351424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829564.834694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333350656:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829564.834710:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829564.834711:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829564.834712:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829564.834715:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829564.834721:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11375 00000100:00080000:0.0:1787829564.834723:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829564.834725:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829564.834728:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829564.834737:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333351552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829564.834750:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.834769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158e300 x1874663333350784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.834784:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333350784/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.834793:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.834797:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.834800:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.834828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829564.834850:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.834864:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158f100 x1874663333350912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.834893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333350912/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.834902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.834906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.834910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333350912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.834918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829564.834939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.834952:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158ca80 x1874663333351040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.834964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333351040/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.834974:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.834977:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.834981:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829564.834987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829564.835007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.835020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158df80 x1874663333351168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.835033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333351168/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.835041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.835044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.835048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829564.835054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829564.835074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.835086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158e680 x1874663333351296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.835099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333351296/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.835107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.835110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.835114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.835120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829564.835139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.835151:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158f800 x1874663333351424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.835162:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333351424/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.835176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.835180:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.835183:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.835189:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829564.835209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829564.835264:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff9057f158c000 x1874663333351552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.835276:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333351552/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.835284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.835287:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829564.835289:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333351552:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829565.218463:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829566.242427:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829567.266422:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829568.290384:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829569.314430:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829569.954355:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954367:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829569.954373:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954384:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954396:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954404:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829569.954450:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333351680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.954485:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954495:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829569.954497:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954502:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954507:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954513:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954524:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829569.954532:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333351680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829569.954542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333351808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.954556:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954560:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.954562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954565:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954569:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954574:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954580:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829569.954594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333351936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.954605:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954609:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829569.954611:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954615:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954619:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954624:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954630:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829569.954643:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333352064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829569.954650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829569.954662:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954665:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.954668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954674:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954678:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954691:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829569.954698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de0700 x1874663333351680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829569.954701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829569.954720:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333352192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829569.954728:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333351680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829569.954737:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829569.954742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829569.954743:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829569.954748:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829569.954750:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829569.954757:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00100000:1.0:1787829569.954758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333351680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829569.954767:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954774:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954779:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829569.954786:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829569.954793:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333352320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.954807:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829569.954813:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829569.954814:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.954819:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.954825:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954829:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829569.954830:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de1500 x1874663333351808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829569.954837:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954862:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829569.954864:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333351808/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.954877:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829569.954879:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333352448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829569.954888:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.954896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351808:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829569.954897:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.954907:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.954910:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829569.954914:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351936:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829569.954918:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11380 00000100:00080000:2.0:1787829569.954925:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.954930:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829569.954936:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829569.954949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829569.954952:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333352576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829569.954966:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de1c00 x1874663333351936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.954981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333351936/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.954989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.954993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.954996:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333351936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829569.955001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829569.955021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829569.955032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de0000 x1874663333352064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.955043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333352064/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.955050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.955053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.955057:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829569.955062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829569.955081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829569.955091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de0a80 x1874663333352192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.955102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333352192/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.955110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.955113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.955117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829569.955134:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829569.955158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829569.955166:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de3800 x1874663333352320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.955174:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333352320/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.955180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.955182:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.955184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829569.955188:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829569.955200:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829569.955206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de1880 x1874663333352448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.955213:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333352448/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.955218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.955242:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.955244:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829569.955248:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829569.955261:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829569.955269:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff9057d0de1180 x1874663333352576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.955277:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333352576/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.955282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.955284:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829569.955287:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333352576:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829570.338461:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829571.362415:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829572.386398:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829573.410431:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829574.434395:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829575.074330:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075435:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829575.075439:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075449:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075465:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075473:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829575.075529:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333352704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829575.075563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075566:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075570:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075573:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829575.075594:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333352832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075602:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075605:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829575.075608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829575.075609:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333352704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829575.075613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075620:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075622:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075626:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829575.075635:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333352960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075646:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075649:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829575.075651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075654:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075656:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075659:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075662:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829575.075671:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333353088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075678:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075680:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829575.075682:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075684:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075687:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075689:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075692:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829575.075712:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333353216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075720:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829575.075724:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352832:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829575.075725:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829575.075729:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075734:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075738:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075740:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075744:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829575.075755:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333353344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829575.075762:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075765:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829575.075766:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352960:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829575.075768:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075776:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075779:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075782:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829575.075786:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829575.075798:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333353472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829575.075822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353088:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787829575.075824:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829575.075829:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829575.075832:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829575.075838:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11385 00000100:00080000:2.0:1787829575.075843:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829575.075849:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829575.075851:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353216:192.168.203.156@tcp:8:kworker.0 00000008:00100000:2.0:1787829575.075859:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829575.075871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829575.075877:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333353600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829575.075890:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829575.075907:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353600:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829575.458427:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829576.482400:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829577.506529:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829578.146507:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829578.146568:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829578.146589:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146647:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146658:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146666:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146675:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146683:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146692:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829578.146701:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787829578.146816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de0a80 x1874663333353600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.146845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333353600/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.146862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.146879:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.146885:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829578.146896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de1180 x1874663333353216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.146921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333353216/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.146931:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.146935:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.146939:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829578.146947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de1880 x1874663333353344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.146961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333353344/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.146970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.146974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.146978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829578.146987:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de3800 x1874663333353472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.147001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333353472/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.147010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.147014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.147018:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333353472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829578.147025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de2a00 x1874663333352832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.147040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333352832/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.147050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.147054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.147058:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829578.147066:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de3100 x1874663333352960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.147080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333352960/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.147089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.147092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.147096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333352960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829578.147104:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de1f80 x1874663333353088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.147119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333353088/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.147127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.147131:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.147135:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829578.147142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829578] req@ffff9057d0de0e00 x1874663333352704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829578.147158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333352704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829578.147167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829578.147171:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829578.147175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333352704:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1787829578.530506:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829579.554356:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829580.194294:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194304:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829580.194308:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194318:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194328:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829580.194335:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829580.194370:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333353728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829580.194394:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194397:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829580.194399:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194402:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194405:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829580.194409:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194419:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829580.194430:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333353856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829580.194440:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194443:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829580.194445:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194452:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00100000:1.0:1787829580.194454:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333353728:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829580.194456:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829580.194468:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194475:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829580.194496:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333353984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829580.194508:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194512:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829580.194515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194520:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194525:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829580.194529:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829580.194552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333354112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829580.194557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829580.194574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194577:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829580.194579:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194585:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194590:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829580.194593:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194598:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829580.194610:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333354240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829580.194631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de0000 x1874663333353728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.194633:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829580.194658:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194669:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194674:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829580.194681:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333353728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.194683:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194699:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829580.194705:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.194716:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829580.194723:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333354368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829580.194731:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333353728:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829580.194746:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194752:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829580.194755:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194761:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194765:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829580.194789:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353856:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829580.194797:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829580.194810:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829580.194832:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333354496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829580.194848:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829580.194860:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829580.194867:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829580.194869:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829580.194874:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11390 00000100:00080000:2.0:1787829580.194881:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829580.194890:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de1c00 x1874663333353856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.194891:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829580.194915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333353856/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829580.194919:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829580.194934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.194944:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829580.194952:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333354624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829580.194955:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829580.194983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829580.195023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de1500 x1874663333353984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333353984/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195061:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195064:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333353984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829580.195071:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829580.195145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195157:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de0700 x1874663333354112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195170:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333354112/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829580.195193:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829580.195217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195271:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de0e00 x1874663333354240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195286:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333354240/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195300:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195305:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829580.195312:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829580.195335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195349:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de1f80 x1874663333354368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195362:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333354368/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195375:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195378:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829580.195403:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829580.195427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195439:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de3100 x1874663333354496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195461:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333354496/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195475:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195480:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829580.195493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829580.195515:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829580.195527:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9057d0de2a00 x1874663333354624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.195539:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333354624/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.195547:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.195550:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829580.195554:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333354624:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829580.578452:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829581.602381:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829582.626409:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829583.650361:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829584.674518:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829585.314347:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314356:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829585.314359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314369:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314378:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314385:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829585.314424:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333354752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314453:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829585.314455:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314459:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314461:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314472:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829585.314482:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333354880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314491:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829585.314496:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314501:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00100000:1.0:1787829585.314502:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333354752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829585.314503:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314510:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829585.314522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333355008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829585.314533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314535:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314538:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314540:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314543:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829585.314552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333355136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314562:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829585.314563:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314566:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314568:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314573:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829585.314574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314577:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829585.314587:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333355264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829585.314597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829585.314598:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314601:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314604:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314607:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314611:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829585.314614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de3800 x1874663333354752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829585.314622:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333355392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314631:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829585.314636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333354752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829585.314637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829585.314640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829585.314646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314648:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314650:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314652:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829585.314662:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333355520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829585.314689:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829585.314690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829585.314695:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829585.314698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829585.314700:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829585.314704:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333354752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829585.314706:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11396 00000100:00080000:2.0:1787829585.314712:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829585.314715:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829585.314717:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829585.314724:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333355648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829585.314731:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829585.314765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314778:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de1880 x1874663333354880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314786:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333354880/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.314792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.314794:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.314797:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333354880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829585.314801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829585.314812:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de1180 x1874663333355008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333355008/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.314830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.314832:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.314834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829585.314838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829585.314849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de0a80 x1874663333355136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333355136/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.314867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.314868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.314870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829585.314873:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829585.314883:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314889:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de2a00 x1874663333355264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333355264/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.314900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.314902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.314903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829585.314906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829585.314917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de3100 x1874663333355392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314931:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333355392/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.314939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.314942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.314945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829585.314951:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829585.314971:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.314983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de1f80 x1874663333355520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.314995:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333355520/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.315001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.315003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.315005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829585.315019:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829585.315033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829585.315039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829585/real 1787829585] req@ffff9057d0de0e00 x1874663333355648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829585.315047:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333355648/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829585.315056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829585.315058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829585.315060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333355648:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829585.698405:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829586.722392:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829587.746389:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829588.770450:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829589.794433:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829590.434365:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434377:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829590.434381:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434394:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434406:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434425:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829590.434471:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333355776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829590.434509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829590.434516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434521:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434525:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434530:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829590.434542:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829590.434557:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333355776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829590.434558:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333355904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829590.434582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829590.434588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434592:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434597:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434601:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829590.434607:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829590.434622:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333356032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829590.434628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829590.434638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434642:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829590.434644:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434648:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434652:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829590.434662:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829590.434673:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333356160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829590.434683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829590.434689:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434694:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434713:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434719:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829590.434724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829590.434739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de0700 x1874663333355776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829590.434740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333356288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829590.434774:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434778:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829590.434781:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829590.434783:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333355776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829590.434789:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:1.0:1787829590.434798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829590.434798:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829590.434806:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829590.434808:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829590.434812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333355776:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787829590.434817:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829590.434832:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333356416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829590.434837:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355904:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829590.434854:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434858:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829590.434860:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829590.434865:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829590.434867:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434874:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434879:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829590.434881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de1500 x1874663333355904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829590.434886:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829590.434894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333355904/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.434902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829590.434904:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333356544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829590.434910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.434920:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333355904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829590.434934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356032:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829590.434938:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829590.434943:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829590.434946:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829590.434951:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11401 00000100:00080000:2.0:1787829590.434956:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829590.434961:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829590.434964:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829590.434972:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829590.434979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de1c00 x1874663333356032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829590.434988:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333356672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829590.434999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333356032/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829590.435017:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829590.435035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829590.435042:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de0000 x1874663333356160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829590.435050:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333356160/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435057:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435059:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829590.435062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829590.435073:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829590.435079:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de0e00 x1874663333356288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829590.435086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333356288/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435094:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829590.435106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829590.435117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829590.435130:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de1f80 x1874663333356416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829590.435138:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333356416/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435145:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435147:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829590.435150:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829590.435161:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829590.435171:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de3100 x1874663333356544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829590.435180:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333356544/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435195:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829590.435201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829590.435252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829590.435263:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829590/real 1787829590] req@ffff9057d0de2a00 x1874663333356672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829590.435271:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333356672/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829590.435277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829590.435279:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829590.435281:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333356672:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829590.818419:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829591.842464:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829592.866476:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829593.890407:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829594.914464:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829595.554287:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554300:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829595.554304:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554315:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554326:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554340:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829595.554373:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333356800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829595.554401:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554405:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829595.554407:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554411:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554415:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554420:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829595.554430:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829595.554440:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333356928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829595.554445:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333356800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829595.554457:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554460:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829595.554461:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554465:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554467:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554470:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829595.554474:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829595.554482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333357056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829595.554488:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829595.554492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554494:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554497:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554498:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829595.554501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829595.554507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829595.554512:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333357184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829595.554534:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829595.554540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554544:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554548:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554552:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554557:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de0a80 x1874663333356800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829595.554559:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829595.554578:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333357312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829595.554579:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333356800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554607:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829595.554609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829595.554613:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829595.554614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829595.554617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829595.554619:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333356800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829595.554623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554627:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554630:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829595.554634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829595.554643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333357440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829595.554645:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356928:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829595.554659:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829595.554677:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829595.554679:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829595.554682:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554686:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829595.554690:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829595.554691:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829595.554697:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829595.554706:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de1180 x1874663333356928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829595.554709:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333357568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829595.554730:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829595.554733:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333356928/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829595.554734:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829595.554738:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829595.554746:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829595.554747:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11406 00000100:00080000:2.0:1787829595.554752:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829595.554756:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829595.554762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:2.0:1787829595.554762:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829595.554767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333356928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829595.554773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829595.554780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333357696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829595.554789:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de1880 x1874663333357056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.554809:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333357056/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554814:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.554817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.554819:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829595.554822:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829595.554832:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de3800 x1874663333357184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.554844:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333357184/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554848:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.554849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.554851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829595.554854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829595.554866:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de2a00 x1874663333357312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.554885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333357312/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.554896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.554899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829595.554915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829595.554928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554935:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de3100 x1874663333357440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.554941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333357440/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554950:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.554952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.554953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829595.554957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829595.554966:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.554971:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de1f80 x1874663333357568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.554977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333357568/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.554981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.554982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.554984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829595.554987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829595.554995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829595.555000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829595/real 1787829595] req@ffff9057d0de0e00 x1874663333357696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829595.555006:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333357696/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829595.555011:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829595.555012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829595.555014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333357696:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829595.938369:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829596.962424:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829597.986417:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829599.010497:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829600.034446:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829600.674332:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829600.674345:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674353:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674364:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674375:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829600.674408:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333357824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674432:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674435:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829600.674437:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674439:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674442:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674445:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674457:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829600.674466:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333357952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674473:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674476:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829600.674477:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674480:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674482:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674484:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674486:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829600.674495:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333357824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829600.674495:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333358080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674508:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674510:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829600.674511:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674513:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674515:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674517:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674521:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829600.674530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333358208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829600.674539:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674541:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674543:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674545:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674547:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829600.674554:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333358336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674560:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829600.674561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829600.674565:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829600.674567:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674571:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674574:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829600.674576:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829600.674586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333358464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829600.674594:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829600.674596:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829600.674599:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674602:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829600.674603:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de0000 x1874663333357824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829600.674608:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829600.674639:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333357824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829600.674640:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333358592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829600.674663:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829600.674666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829600.674668:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829600.674671:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829600.674671:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829600.674677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333357824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829600.674680:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11411 00000100:00080000:2.0:1787829600.674686:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829600.674687:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357952:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829600.674695:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829600.674701:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829600.674710:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829600.674716:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333358720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829600.674721:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de1c00 x1874663333357952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333357952/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674744:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674746:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333357952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.674768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829600.674780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de1500 x1874663333358080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674793:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333358080/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674797:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674799:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674801:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.674804:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829600.674814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674819:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de0700 x1874663333358208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333358208/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674829:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674831:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674833:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829600.674835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829600.674845:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de0e00 x1874663333358336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333358336/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674865:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674867:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829600.674869:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829600.674878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674883:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de1f80 x1874663333358464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333358464/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674893:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.674899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829600.674908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674913:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de3100 x1874663333358592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674919:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333358592/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.674933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829600.674942:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829600.674946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff9057d0de2a00 x1874663333358720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.674952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333358720/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.674956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.674958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829600.674960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333358720:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829601.058388:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829602.082395:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829603.106381:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829604.130391:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829605.154419:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829605.794312:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794323:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829605.794327:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794340:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794352:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794367:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829605.794408:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333358848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794433:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794437:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829605.794439:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794442:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794446:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794450:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794459:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829605.794469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333358848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829605.794471:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333358976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794483:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794486:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829605.794487:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794491:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794494:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794496:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794499:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829605.794508:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333359104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794517:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829605.794518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794521:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794524:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794527:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794531:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829605.794537:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333359232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794543:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794546:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829605.794547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794549:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794551:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794553:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794555:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829605.794560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333359360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794565:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794567:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829605.794568:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794571:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794573:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829605.794576:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829605.794579:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794584:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829605.794590:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333359488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794614:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794617:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829605.794619:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829605.794622:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794626:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794629:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794634:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829605.794640:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78a9f80 x1874663333358848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829605.794646:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333359616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829605.794664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829605.794667:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829605.794670:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829605.794671:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333358848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.794675:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11416 00000100:00080000:1.0:1787829605.794680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829605.794686:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829605.794691:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829605.794693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829605.794701:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333359744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829605.794705:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333358848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829605.794734:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829605.794753:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794762:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78a8000 x1874663333358976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794771:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333358976/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794776:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794778:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794781:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333358976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829605.794784:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829605.794794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78a9880 x1874663333359104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333359104/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794813:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794814:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794816:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829605.794820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829605.794832:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794837:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78ab800 x1874663333359232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333359232/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829605.794857:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829605.794867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78ab480 x1874663333359360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794880:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333359360/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794885:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794886:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333359360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829605.794891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829605.794902:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794908:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78aaa00 x1874663333359488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794916:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333359488/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829605.794928:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829605.794946:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.794953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78a8380 x1874663333359616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.794960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333359616/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.794965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.794967:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.794969:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829605.794973:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829605.794983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829605.795001:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff9057c78aad80 x1874663333359744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829605.795009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333359744/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829605.795021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829605.795023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829605.795025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333359744:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829606.178456:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829607.202423:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829608.226404:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829609.250388:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829610.274447:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829610.914328:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914336:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829610.914340:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914363:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914369:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829610.914404:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333359872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914429:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829610.914430:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914436:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914438:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829610.914453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333360000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829610.914468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914474:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829610.914482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333360128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914488:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829610.914492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914494:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914496:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914498:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829610.914507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333360256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914515:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829610.914516:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914519:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914521:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914523:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914527:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829610.914531:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333359872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829610.914535:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333360384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914546:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829610.914549:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914552:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914554:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914573:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829610.914587:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333360512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914594:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914596:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829610.914598:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914600:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914605:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914608:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829610.914618:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829610.914619:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333360640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829610.914633:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829610.914635:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829610.914636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829610.914639:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11421 00000100:00080000:3.0:1787829610.914641:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829610.914643:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829610.914646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829610.914653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333360768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829610.914669:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb223800 x1874663333359872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.914691:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333359872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.914702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.914707:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.914711:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333359872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829610.914719:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829610.914744:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.914758:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb220000 x1874663333360000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.914771:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333360000/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.914779:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.914783:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.914787:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829610.914794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829610.914863:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.914875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb220a80 x1874663333360128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.914888:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333360128/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.914897:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.914900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.914904:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829610.914910:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829610.914928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.914939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb223b80 x1874663333360256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.914951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333360256/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.914959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.914962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.914966:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829610.914992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829610.915012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.915023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb221180 x1874663333360384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.915034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333360384/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.915043:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.915047:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.915050:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333360384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829610.915056:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829610.915075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.915087:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb222680 x1874663333360512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.915099:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333360512/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.915107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.915110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.915114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829610.915120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829610.915139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.915150:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb220e00 x1874663333360640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.915163:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333360640/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.915172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.915175:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.915178:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829610.915185:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829610.915205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829610.915218:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff9057fb221500 x1874663333360768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829610.915258:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333360768/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829610.915267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829610.915270:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829610.915274:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333360768:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829611.298385:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829612.322387:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829613.346423:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829614.370392:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829615.394453:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829616.034313:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034340:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829616.034350:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829616.034358:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034368:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829616.034374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829616.034406:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333360896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034431:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829616.034435:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829616.034438:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034440:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829616.034443:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034451:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829616.034461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333361024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829616.034473:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829616.034476:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034480:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.034481:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333360896:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829616.034491:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829616.034508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333361152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034516:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829616.034532:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829616.034536:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034545:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829616.034549:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829616.034554:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034556:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829616.034568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333361280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829616.034580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829616.034584:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034587:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.034591:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb222a00 x1874663333360896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829616.034594:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034601:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829616.034615:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333361408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829616.034618:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333360896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829616.034637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829616.034641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829616.034642:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829616.034649:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829616.034651:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034655:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.034656:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333360896:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829616.034661:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829616.034667:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829616.034676:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333361536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034688:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829616.034690:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829616.034696:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829616.034699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829616.034703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb220700 x1874663333361024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829616.034708:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034718:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829616.034726:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829616.034730:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333361024/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829616.034734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829616.034741:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.034746:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.034750:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829616.034756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333361664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829616.034770:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829616.034773:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829616.034776:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829616.034783:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361152:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829616.034783:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11426 00000100:00080000:3.0:1787829616.034789:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829616.034794:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829616.034799:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829616.034811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333361792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829616.034818:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.034839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb222300 x1874663333361152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.034848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333361152/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.034855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.034858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.034860:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.034865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.034897:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.034906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb220380 x1874663333361280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.034913:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333361280/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.034919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.034921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.034924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829616.034927:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.034938:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.034944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb221880 x1874663333361408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.034951:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333361408/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.034957:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.034958:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.034960:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333361408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829616.034963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829616.034973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.034979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb223100 x1874663333361536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.034991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333361536/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.034997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.034998:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.035000:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.035003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829616.035013:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.035019:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb222d80 x1874663333361664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.035027:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333361664/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.035031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.035033:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.035035:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.035048:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.035058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829616.035064:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9057fb221c00 x1874663333361792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.035072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333361792/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.035081:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.035083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829616.035085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333361792:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829616.418386:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829617.442388:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829618.466345:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829619.490384:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829620.514439:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829621.154331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154342:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829621.154347:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829621.154356:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829621.154411:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333361920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829621.154433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829621.154437:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829621.154440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154443:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154454:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829621.154465:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333362048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829621.154470:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333361920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829621.154478:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154482:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829621.154483:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829621.154501:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154505:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154508:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154513:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829621.154523:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333362176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829621.154530:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829621.154534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829621.154537:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11431 00000800:00000100:2.0:1787829621.154542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829621.154543:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154548:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154552:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829621.154560:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333362304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829621.154566:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154568:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829621.154570:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829621.154573:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154576:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829621.154580:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78abb80 x1874663333361920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829621.154588:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829621.154597:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333361920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.154609:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829621.154614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333362432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829621.154615:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.154622:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333361920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829621.154637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154641:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829621.154643:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829621.154647:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362048:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829621.154649:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154657:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154660:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829621.154672:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829621.154679:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333362560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829621.154685:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78aa680 x1874663333362048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.154707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333362048/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829621.154708:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154719:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829621.154721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829621.154723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829621.154726:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829621.154731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00100000:2.0:1787829621.154732:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362048:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829621.154738:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829621.154742:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362176:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829621.154746:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829621.154752:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829621.154761:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829621.154766:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333362688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829621.154772:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78a9c00 x1874663333362176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829621.154777:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829621.154786:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829621.154788:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829621.154791:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333362176/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829621.154794:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11431 00000100:00080000:1.0:1787829621.154799:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829621.154806:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829621.154807:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829621.154810:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829621.154813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.154817:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829621.154822:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333362816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829621.154827:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.154856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829621.154869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78aa300 x1874663333362304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.154883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333362304/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.154892:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.154895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.154898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829621.154903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.154918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829621.154926:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78a8a80 x1874663333362432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.154933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333362432/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.154939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.154941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.154943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333362432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829621.154954:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829621.154965:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829621.154971:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78a9180 x1874663333362560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.154981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333362560/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.154986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.154989:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.154991:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.154996:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829621.155016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829621.155023:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78a8700 x1874663333362688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.155030:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333362688/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.155036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.155038:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.155040:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.155043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.155054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829621.155067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9057c78a9500 x1874663333362816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.155075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333362816/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.155081:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.155084:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829621.155086:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333362816:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829621.538459:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829622.562336:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829623.586410:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829624.610373:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829625.634408:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829626.274343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274352:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829626.274356:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274373:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274380:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829626.274437:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333362944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829626.274515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274519:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274522:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274526:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829626.274547:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333363072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274557:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829626.274559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333362944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829626.274560:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829626.274564:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274571:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274573:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829626.274588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333363200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274596:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274598:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829626.274600:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274602:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274605:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274607:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829626.274618:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333363328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829626.274634:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274636:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274638:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274640:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274643:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829626.274649:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333363456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274658:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829626.274659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274662:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274664:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274666:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829626.274674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333363584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274680:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829626.274686:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829626.274689:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829626.274692:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274696:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274698:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274701:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829626.274708:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333363712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829626.274716:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829626.274718:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829626.274720:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829626.274722:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11437 00000100:00080000:3.0:1787829626.274725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829626.274728:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829626.274732:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829626.274745:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333363840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829626.274746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb221c00 x1874663333362944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.274794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333362944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.274807:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.274812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.274816:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333362944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829626.274856:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829626.274892:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.274905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb222d80 x1874663333363072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.274917:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333363072/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.274924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.274927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.274950:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829626.274956:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829626.274970:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.274977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb223100 x1874663333363200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.274986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333363200/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.274991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.274993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.274995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829626.274999:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829626.275012:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.275018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb221880 x1874663333363328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.275026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333363328/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.275032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.275034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.275036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829626.275040:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829626.275050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.275056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb220380 x1874663333363456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.275064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333363456/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.275069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.275072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.275074:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333363456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829626.275079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829626.275098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.275110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb222300 x1874663333363584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.275158:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333363584/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.275164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.275166:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.275169:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829626.275173:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829626.275205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.275212:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb220700 x1874663333363712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.275220:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333363712/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.275225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.275227:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.275228:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829626.275232:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829626.275285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829626.275293:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829626/real 1787829626] req@ffff9057fb222a00 x1874663333363840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829626.275301:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333363840/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829626.275312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829626.275314:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829626.275316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333363840:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829626.658435:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829627.682475:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829628.706379:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829629.730385:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829630.754375:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829631.394347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394358:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829631.394361:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394383:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829631.394391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829631.394441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333363968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829631.394470:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394474:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829631.394476:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394481:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394486:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829631.394491:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829631.394501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829631.394507:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333363968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829631.394522:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333364096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829631.394536:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394540:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829631.394543:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394547:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394552:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829631.394556:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829631.394561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829631.394566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829631.394582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333364224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829631.394593:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394597:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829631.394600:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829631.394605:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78ab100 x1874663333363968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829631.394609:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394619:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829631.394625:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829631.394628:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333363968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829631.394636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829631.394639:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.394643:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.394646:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333363968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829631.394653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333364352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829631.394665:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829631.394667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394673:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829631.394677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394683:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394687:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829631.394693:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829631.394697:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829631.394713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829631.394713:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78a9500 x1874663333364096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829631.394729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333364096/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829631.394737:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333364480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829631.394738:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.394746:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.394752:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829631.394762:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364224:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829631.394764:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394770:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829631.394772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394778:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11442 00000800:00000100:2.0:1787829631.394782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829631.394786:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829631.394794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78a8700 x1874663333364224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829631.394796:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829631.394808:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829631.394820:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333364224/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829631.394825:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333364608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829631.394831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.394837:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.394843:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364224:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829631.394853:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829631.394858:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829631.394861:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364352:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829631.394862:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394870:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00080000:1.0:1787829631.394876:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829631.394881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829631.394885:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829631.394891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78a9180 x1874663333364352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829631.394893:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829631.394912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333364352/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829631.394914:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333364736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829631.394937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829631.394943:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829631.394945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829631.394950:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829631.394955:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364352:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829631.394956:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829631.394967:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11442 00000100:00100000:2.0:1787829631.394968:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364480:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829631.394975:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829631.394983:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829631.394989:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829631.394993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829631.395007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78a8a80 x1874663333364480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829631.395009:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333364864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829631.395024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333364480/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829631.395037:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.395040:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.395043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333364480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829631.395048:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829631.395061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829631.395068:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78aa300 x1874663333364608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829631.395075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333364608/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829631.395080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.395082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.395084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829631.395097:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829631.395116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829631.395128:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78a9c00 x1874663333364736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829631.395146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333364736/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829631.395152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.395154:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.395157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829631.395161:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829631.395192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829631.395200:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829631/real 1787829631] req@ffff9057c78aa680 x1874663333364864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829631.395208:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333364864/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829631.395214:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829631.395216:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829631.395218:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333364864:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829631.778359:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829632.802460:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829633.826416:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829634.850519:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829635.874373:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829636.514325:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514336:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829636.514341:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514352:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514365:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829636.514437:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333364992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829636.514461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514464:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829636.514465:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514469:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514473:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514477:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829636.514486:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829636.514501:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333365120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829636.514503:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333364992:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829636.514529:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514532:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829636.514533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514538:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514540:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514545:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829636.514549:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829636.514557:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333365248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829636.514575:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829636.514584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829636.514591:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829636.514597:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514606:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514611:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829636.514619:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829636.514640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333365376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829636.514641:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78abb80 x1874663333364992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.514683:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514686:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829636.514687:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514702:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514708:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829636.514715:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333364992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.514716:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829636.514734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829636.514736:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829636.514744:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.514749:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333364992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829636.514755:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333365504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829636.514765:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514767:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829636.514769:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514775:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00100000:2.0:1787829636.514781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365120:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829636.514782:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514796:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829636.514802:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829636.514811:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333365632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829636.514821:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829636.514829:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829636.514835:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829636.514838:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514845:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00100000:2.0:1787829636.514847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78aad80 x1874663333365120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.514852:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829636.514862:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829636.514875:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829636.514878:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333365120/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.514890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829636.514894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333365760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829636.514898:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.514913:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365120:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829636.514931:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829636.514934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365248:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829636.514937:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829636.514942:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829636.514949:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11447 00000100:00080000:1.0:1787829636.514955:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829636.514958:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829636.514966:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829636.514973:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78a8380 x1874663333365248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829636.514976:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829636.514995:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333365248/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.515000:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333365888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829636.515038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515042:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515046:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829636.515054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829636.515069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829636.515081:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78aaa00 x1874663333365376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829636.515089:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333365376/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.515094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515096:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515099:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829636.515102:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829636.515113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829636.515125:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78ab480 x1874663333365504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829636.515133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333365504/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.515138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515140:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515142:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333365504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829636.515155:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829636.515166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829636.515173:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78ab800 x1874663333365632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829636.515185:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333365632/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.515190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515194:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829636.515197:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829636.515208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829636.515214:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78a9880 x1874663333365760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829636.515257:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333365760/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.515262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515264:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515266:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829636.515271:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829636.515283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829636.515290:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829636/real 1787829636] req@ffff9057c78a8000 x1874663333365888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829636.515298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333365888/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829636.515307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829636.515309:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829636.515311:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333365888:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829636.898424:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829637.922385:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829638.946431:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829639.970364:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829640.994396:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829641.634292:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634301:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829641.634305:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634313:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634323:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634331:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829641.634382:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333366016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829641.634410:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634414:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829641.634416:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634420:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634424:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634428:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829641.634437:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829641.634453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333366144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829641.634457:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333366016:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829641.634474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634479:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829641.634481:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634486:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634490:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634495:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829641.634500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829641.634513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333366272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829641.634523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634526:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829641.634528:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634533:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634537:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634542:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829641.634545:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366144:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829641.634551:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829641.634567:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333366400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829641.634575:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829641.634587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829641.634594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634601:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00100000:2.0:1787829641.634602:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366400:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829641.634607:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634614:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829641.634621:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829641.634634:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333366528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829641.634652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829641.634658:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634663:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634667:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634674:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829641.634675:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366528:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1787829641.634682:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829641.634707:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333366656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829641.634722:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634726:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829641.634728:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829641.634733:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00100000:2.0:1787829641.634736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366656:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829641.634739:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634746:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829641.634752:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829641.634766:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333366784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829641.634782:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829641.634786:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829641.634789:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829641.634793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829641.634795:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11452 00000100:00080000:1.0:1787829641.634802:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829641.634807:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829641.634814:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829641.634828:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333366912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829641.634881:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366912:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829642.018447:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829643.042370:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829644.066365:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829644.706691:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787829644.706739:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829644.706753:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706797:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706806:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706815:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706823:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706831:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706839:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829644.706858:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829644.706891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78aad80 x1874663333366912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.706912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333366912/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.706923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.706928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.706934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829644.706942:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78a8380 x1874663333366784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.706955:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333366784/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.706963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.706966:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.706970:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829644.706976:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78aaa00 x1874663333366656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333366656/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707008:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707011:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707014:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333366656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829644.707021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78ab480 x1874663333366528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333366528/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829644.707054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78ab800 x1874663333366400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707066:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333366400/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707082:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829644.707089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78a8000 x1874663333366144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707102:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333366144/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707118:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829644.707125:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78a9880 x1874663333366272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707137:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333366272/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707148:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333366272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829644.707159:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829644] req@ffff9057c78a9f80 x1874663333366016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829644.707171:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333366016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829644.707178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829644.707181:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829644.707184:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333366016:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1787829645.090394:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829646.114377:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829646.754353:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754362:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829646.754366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754375:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829646.754390:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829646.754419:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333367040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754453:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829646.754455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829646.754467:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829646.754476:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829646.754492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333367168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754503:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754507:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829646.754509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754518:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829646.754521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829646.754526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829646.754531:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333367040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829646.754540:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333367296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829646.754558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754561:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754567:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829646.754571:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829646.754577:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829646.754588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333367424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829646.754599:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829646.754605:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829646.754608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754614:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754627:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829646.754632:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829646.754638:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829646.754644:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb221500 x1874663333367040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829646.754653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333367552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754667:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829646.754672:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333367040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829646.754673:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829646.754676:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754685:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754692:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829646.754694:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829646.754698:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829646.754704:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787829646.754707:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829646.754710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333367040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829646.754724:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829646.754724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333367680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754741:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754745:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829646.754748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754753:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754758:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829646.754761:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829646.754766:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829646.754774:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829646.754783:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb220e00 x1874663333367168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829646.754790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333367808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829646.754820:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829646.754825:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829646.754829:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333367168/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829646.754830:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829646.754838:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11457 00000100:00080000:3.0:1787829646.754845:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829646.754849:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829646.754853:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829646.754858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787829646.754862:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829646.754865:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829646.754879:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333367936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829646.754897:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829646.754921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.754933:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb222680 x1874663333367296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.754946:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333367296/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.754955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.754959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.754962:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829646.754969:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829646.754990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.755002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb221180 x1874663333367424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.755014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333367424/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.755029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.755032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.755036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829646.755042:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829646.755061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.755072:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb223b80 x1874663333367552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.755083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333367552/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.755091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.755095:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.755098:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333367552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829646.755105:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829646.755124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.755136:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb220a80 x1874663333367680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.755146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333367680/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.755153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.755156:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.755159:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829646.755165:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829646.755184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.755195:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb220000 x1874663333367808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.755207:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333367808/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.755214:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.755217:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.755253:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829646.755260:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829646.755280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829646.755291:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff9057fb223800 x1874663333367936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.755302:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333367936/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.755328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.755332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829646.755335:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333367936:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829647.138387:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829648.162446:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829649.186505:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829650.210420:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829651.234386:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829651.874308:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874316:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829651.874320:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874329:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874338:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874344:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829651.874376:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333368064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874399:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874402:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829651.874403:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874406:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874408:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874410:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874417:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829651.874426:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333368192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874434:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874436:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829651.874439:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829651.874441:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333368064:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829651.874444:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874449:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874453:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874455:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829651.874464:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333368320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874471:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874474:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829651.874475:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874477:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874479:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874482:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874485:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829651.874492:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333368448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829651.874501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874503:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874505:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874507:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874510:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829651.874517:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333368576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874522:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874524:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829651.874525:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874528:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874530:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874535:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829651.874541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333368704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829651.874548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829651.874550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829651.874554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874557:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874559:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874561:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874564:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829651.874570:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333368832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829651.874577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829651.874578:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829651.874579:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829651.874581:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11462 00000100:00080000:2.0:1787829651.874584:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829651.874586:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829651.874589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829651.874597:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333368960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829651.874604:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de3800 x1874663333368064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874628:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333368064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874643:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333368064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829651.874661:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829651.874677:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874686:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de1880 x1874663333368192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874694:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333368192/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874706:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874710:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829651.874717:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829651.874737:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874749:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de1180 x1874663333368320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874762:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333368320/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874769:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874772:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829651.874776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829651.874791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874797:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de0a80 x1874663333368448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874805:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333368448/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874810:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829651.874827:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829651.874838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874845:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de2a00 x1874663333368576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874859:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333368576/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333368576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829651.874871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829651.874882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de3100 x1874663333368704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333368704/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874901:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829651.874906:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829651.874916:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874922:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de1f80 x1874663333368832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333368832/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874942:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829651.874946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829651.874955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829651.874960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff9057d0de0e00 x1874663333368960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829651.874967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333368960/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829651.874971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829651.874973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829651.874975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333368960:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829652.258398:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829653.282381:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829654.306413:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829655.330376:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829656.354502:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829656.994334:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829656.994344:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829656.994348:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994359:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994370:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829656.994378:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829656.994414:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333369088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829656.994441:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829656.994446:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829656.994448:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994454:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829656.994463:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829656.994469:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333369088:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787829656.994473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829656.994490:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333369216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829656.994502:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829656.994505:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829656.994507:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994511:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994517:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829656.994522:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829656.994525:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829656.994532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829656.994548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333369344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829656.994558:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829656.994564:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829656.994565:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78abb80 x1874663333369088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829656.994569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994577:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994580:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829656.994591:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829656.994594:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333369088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829656.994602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829656.994605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.994610:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.994614:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333369088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829656.994619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333369472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829656.994630:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829656.994634:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829656.994637:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369216:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829656.994639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994646:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994652:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829656.994657:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829656.994660:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829656.994667:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829656.994673:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78aa680 x1874663333369216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829656.994683:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333369600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829656.994687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333369216/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.994710:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829656.994715:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829656.994715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.994721:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369216:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829656.994723:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829656.994728:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829656.994728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369344:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829656.994738:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994745:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829656.994749:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829656.994752:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829656.994761:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829656.994761:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78a9c00 x1874663333369344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.994775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333369344/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829656.994780:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333369728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829656.994782:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.994788:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.994794:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369344:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829656.994806:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829656.994806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369472:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829656.994814:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829656.994818:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994823:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11467 00000800:00000100:2.0:1787829656.994826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829656.994830:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829656.994838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78aa300 x1874663333369472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829656.994839:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829656.994847:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829656.994862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333369472/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829656.994865:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333369856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829656.994886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829656.994890:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829656.994894:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829656.994897:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829656.994900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369472:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829656.994902:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829656.994909:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11467 00000100:00080000:1.0:1787829656.994914:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829656.994918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369600:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829656.994921:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829656.994928:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829656.994940:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829656.994944:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333369984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829656.994954:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78a8a80 x1874663333369600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.994977:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333369600/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.994987:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.994991:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.994995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333369600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829656.995001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829656.995017:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829656.995025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78a9180 x1874663333369728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.995033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333369728/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.995041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.995044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.995048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829656.995054:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829656.995074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829656.995086:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78a8700 x1874663333369856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.995095:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333369856/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.995105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.995108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.995110:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829656.995120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829656.995134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829656.995141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff9057c78a9500 x1874663333369984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.995148:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333369984/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.995153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.995155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829656.995157:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333369984:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829657.378409:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829658.402393:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829659.426370:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829660.450417:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829661.474348:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829662.114331:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829662.114350:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114361:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114373:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114381:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829662.114415:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333370112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829662.114437:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114440:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829662.114441:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114444:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114447:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114449:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114457:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829662.114467:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333370240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829662.114474:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114477:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829662.114478:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114481:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114482:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114485:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114489:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829662.114491:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333370112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829662.114498:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333370368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829662.114507:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114509:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829662.114510:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114512:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114515:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114517:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114521:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829662.114530:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333370496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829662.114536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114539:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829662.114540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114542:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114545:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114555:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829662.114565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333370624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829662.114573:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829662.114577:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114580:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829662.114581:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114585:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114588:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114591:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114594:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829662.114602:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333370752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829662.114614:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78ab100 x1874663333370112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.114615:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114623:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829662.114627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114634:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114638:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829662.114643:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333370112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.114644:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829662.114650:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829662.114661:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829662.114664:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333370880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.114668:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.114677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333370112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829662.114681:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829662.114683:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829662.114684:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829662.114688:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11472 00000100:00080000:1.0:1787829662.114692:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829662.114695:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829662.114699:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370240:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829662.114700:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829662.114711:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333371008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829662.114730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.114746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78a9500 x1874663333370240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.114769:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333370240/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.114777:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.114779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.114782:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829662.114801:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829662.114821:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.114834:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78a8700 x1874663333370368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.114846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333370368/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.114855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.114858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.114861:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829662.114868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829662.114888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.114900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78a9180 x1874663333370496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.114912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333370496/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.114919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.114922:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.114925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829662.114931:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829662.114949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.114960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78a8a80 x1874663333370624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.114970:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333370624/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.114977:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.114980:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.114983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333370624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829662.114988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333370752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829662.115005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.115015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78aa300 x1874663333370752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.115026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333370752/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.115035:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.115037:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.115041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333370752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829662.115047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333370880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829662.115066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.115076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78a9c00 x1874663333370880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.115084:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333370880/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.115089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.115092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.115096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333370880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829662.115102:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829662.115121:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829662.115133:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9057c78aa680 x1874663333371008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.115146:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333371008/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.115154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.115157:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829662.115162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371008:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829662.498384:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829663.522431:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829664.546462:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829665.570347:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829666.594390:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829667.234330:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829667.234342:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234350:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234365:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829667.234396:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333371136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234415:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234418:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829667.234419:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234423:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234425:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234427:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234433:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829667.234442:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333371264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234450:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234453:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829667.234455:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234458:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234460:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234463:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234466:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829667.234471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333371136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829667.234476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333371392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234485:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234488:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829667.234489:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234491:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234493:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234496:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234498:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829667.234505:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333371520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234511:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234514:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829667.234515:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234518:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234519:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234522:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234525:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829667.234532:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333371648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234538:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234541:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829667.234542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234545:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234547:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829667.234552:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829667.234554:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829667.234565:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333371776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829667.234573:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234575:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829667.234576:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234579:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234581:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234584:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234587:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829667.234596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333371904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829667.234600:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb223800 x1874663333371136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829667.234605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829667.234607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829667.234608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829667.234614:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11477 00000100:00080000:3.0:1787829667.234620:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829667.234629:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829667.234635:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829667.234642:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333371136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829667.234650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333372032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829667.234657:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.234673:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.234678:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333371136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829667.234686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829667.234710:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.234722:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb220000 x1874663333371264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.234734:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333371264/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.234742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.234745:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.234748:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829667.234781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829667.234804:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.234817:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb220a80 x1874663333371392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.234829:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333371392/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.234837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.234840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.234844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829667.234850:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829667.234870:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.234881:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb223b80 x1874663333371520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.234892:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333371520/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.234900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.234903:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.234915:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829667.234920:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829667.234937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.234947:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb221180 x1874663333371648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.234964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333371648/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.234971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.234974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.234978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333371648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829667.234984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829667.235002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.235012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb222680 x1874663333371776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.235024:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333371776/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.235032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.235035:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.235038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829667.235044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829667.235064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.235076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb220e00 x1874663333371904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.235087:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333371904/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.235100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.235104:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.235107:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333371904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829667.235113:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829667.235130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829667.235142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829667/real 1787829667] req@ffff9057fb221500 x1874663333372032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829667.235179:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333372032/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829667.235188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829667.235191:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829667.235194:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372032:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829667.618403:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829668.642393:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829669.666419:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829670.690466:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829671.714341:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829672.354336:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354348:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829672.354352:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354365:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354387:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829672.354429:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333372160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829672.354465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354470:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829672.354472:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354478:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354484:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829672.354512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333372160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829672.354518:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333372288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829672.354537:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354540:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829672.354542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354546:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354550:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354553:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354557:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829672.354568:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333372416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829672.354576:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829672.354577:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829672.354585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829672.354587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354591:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354595:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354599:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354603:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829672.354614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333372544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829672.354622:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829672.354626:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb222a00 x1874663333372160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829672.354627:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829672.354631:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354641:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354646:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354652:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354656:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829672.354664:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333372160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829672.354668:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333372672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829672.354700:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354704:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829672.354706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829672.354708:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829672.354710:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354716:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829672.354722:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829672.354723:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354732:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829672.354734:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333372160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829672.354764:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333372800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829672.354771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372288:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829672.354780:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354785:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829672.354787:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354791:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354803:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829672.354809:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354812:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829672.354816:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829672.354824:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333372928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829672.354836:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb220700 x1874663333372288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829672.354844:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829672.354853:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829672.354856:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829672.354864:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11483 00000100:00080000:3.0:1787829672.354867:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829672.354877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333372288/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829672.354905:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829672.354910:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829672.354922:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829672.354927:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333373056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829672.354929:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.354938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829672.354971:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829672.354993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb222300 x1874663333372416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333372416/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355026:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829672.355030:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829672.355044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355051:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb220380 x1874663333372544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333372544/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355069:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355071:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829672.355075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829672.355088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355095:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb221880 x1874663333372672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355123:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333372672/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355168:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355172:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333372672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829672.355192:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829672.355210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355219:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb223100 x1874663333372800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355275:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333372800/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355285:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355288:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829672.355293:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829672.355310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355318:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb222d80 x1874663333372928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355327:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333372928/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355335:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355338:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333372928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829672.355342:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829672.355354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829672.355361:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829672/real 1787829672] req@ffff9057fb221c00 x1874663333373056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829672.355387:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333373056/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829672.355400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829672.355402:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829672.355405:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373056:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829672.738421:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829673.762351:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829674.786395:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829675.810378:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829676.834396:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829677.474309:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829677.474320:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829677.474324:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474334:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00080000:1.0:1787829677.474345:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474354:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829677.474392:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333373184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829677.474418:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829677.474422:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829677.474424:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474429:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00080000:1.0:1787829677.474433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474438:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474458:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829677.474458:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333373184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829677.474477:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333373312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829677.474496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829677.474500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829677.474502:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474507:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00080000:1.0:1787829677.474512:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474517:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829677.474538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333373440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829677.474538:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829677.474560:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829677.474564:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829677.474566:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474570:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00080000:1.0:1787829677.474574:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474579:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829677.474586:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78abb80 x1874663333373184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829677.474601:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333373568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829677.474606:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333373184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829677.474626:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829677.474630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829677.474632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829677.474637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829677.474638:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829677.474648:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00100000:2.0:1787829677.474649:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333373184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829677.474655:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474662:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474667:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829677.474676:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829677.474683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333373696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829677.474696:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829677.474701:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829677.474704:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829677.474709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474716:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00100000:2.0:1787829677.474717:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78a9f80 x1874663333373312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829677.474723:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474731:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829677.474738:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333373312/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829677.474739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829677.474754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829677.474758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333373824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829677.474762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.474772:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373312:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829677.474777:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829677.474782:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829677.474784:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373440:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829677.474785:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829677.474794:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00080000:1.0:1787829677.474799:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474803:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474808:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829677.474814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.474826:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78a9880 x1874663333373440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829677.474832:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333373952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829677.474839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333373440/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829677.474849:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829677.474851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829677.474856:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829677.474857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829677.474860:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829677.474862:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373440:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829677.474867:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11488 00000100:00100000:2.0:1787829677.474871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373568:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829677.474873:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829677.474880:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829677.474886:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829677.474890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.474897:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78a8000 x1874663333373568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829677.474906:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333374080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829677.474907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333373568/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829677.474934:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829677.474940:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.474943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829677.474958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829677.474969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.474975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78ab800 x1874663333373696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829677.474987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333373696/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829677.474992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829677.474993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.474995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333373696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829677.474999:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829677.475009:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.475014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78ab480 x1874663333373824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829677.475022:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333373824/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829677.475026:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829677.475028:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.475030:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829677.475033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829677.475042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.475047:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78aaa00 x1874663333373952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829677.475054:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333373952/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829677.475063:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829677.475065:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.475067:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333373952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829677.475070:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829677.475079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829677.475084:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829677/real 1787829677] req@ffff9057c78a8380 x1874663333374080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829677.475091:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333374080/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829677.475111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829677.475114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829677.475116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374080:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829677.858447:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829678.882388:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829679.906424:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829680.930359:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829681.954421:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829682.594323:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594331:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829682.594335:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594344:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594353:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594360:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829682.594393:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333374208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.594411:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594414:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829682.594415:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594418:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594421:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594424:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594430:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829682.594441:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333374336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829682.594446:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333374208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829682.594451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594456:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.594457:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594462:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594464:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594467:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829682.594476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333374464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.594483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829682.594487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594489:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594492:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594494:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829682.594505:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333374592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829682.594513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829682.594516:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594519:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.594521:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594525:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594528:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594531:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829682.594542:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333374720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.594547:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.594549:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829682.594550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594553:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594557:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829682.594560:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78aad80 x1874663333374208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829682.594563:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594570:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829682.594585:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333374848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829682.594588:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333374208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829682.594607:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829682.594610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829682.594611:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.594615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829682.594618:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829682.594622:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594626:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829682.594627:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333374208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829682.594632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829682.594643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333374976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.594652:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829682.594652:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374336:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829682.594660:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.594661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.594664:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11493 00000100:00080000:1.0:1787829682.594667:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.594670:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829682.594673:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829682.594678:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829682.594684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333375104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829682.594701:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78a8380 x1874663333374336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333374336/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594719:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594722:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594724:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829682.594728:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829682.594741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78aaa00 x1874663333374464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594761:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333374464/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594769:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594771:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829682.594774:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829682.594786:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78ab480 x1874663333374592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594800:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333374592/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594807:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829682.594813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829682.594823:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594828:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78ab800 x1874663333374720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594835:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333374720/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333374720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829682.594847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829682.594857:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594862:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78a8000 x1874663333374848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333374848/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594876:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594878:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829682.594889:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829682.594899:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78a9880 x1874663333374976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333374976/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594918:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594919:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594921:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333374976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829682.594924:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333375104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829682.594934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829682.594940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829682] req@ffff9057c78a9f80 x1874663333375104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829682.594950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333375104/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829682.594958:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829682.594961:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829682.594965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333375104:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829682.978360:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829684.002380:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829685.026466:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829686.050369:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829687.074507:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829687.714359:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714371:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829687.714376:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714388:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714399:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714406:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829687.714440:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333375232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714466:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829687.714468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714492:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714496:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829687.714515:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333375232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829687.714521:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333375360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714536:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714540:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829687.714542:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714546:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714550:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714555:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714560:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829687.714571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333375488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714581:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829687.714582:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714587:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11498 00000800:00000100:2.0:1787829687.714590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829687.714592:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714596:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714599:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829687.714608:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333375616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714617:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829687.714618:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714621:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714623:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714626:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714629:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829687.714632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78abb80 x1874663333375232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.714641:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333375744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714648:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829687.714651:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333375232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.714652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829687.714656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714666:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:2.0:1787829687.714670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.714671:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714679:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829687.714680:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829687.714685:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829687.714686:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333375232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829687.714695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333375872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714702:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714704:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829687.714706:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714708:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714710:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714712:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829687.714723:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829687.714724:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333376000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829687.714738:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829687.714740:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829687.714741:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829687.714744:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11498 00000100:00080000:1.0:1787829687.714747:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829687.714749:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829687.714753:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829687.714759:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829687.714765:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333376128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829687.714778:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78aa680 x1874663333375360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.714794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333375360/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.714804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.714816:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.714820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829687.714828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829687.714851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.714863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78a9c00 x1874663333375488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.714872:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333375488/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.714894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.714897:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.714899:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829687.714903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829687.714918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.714925:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78aa300 x1874663333375616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.714933:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333375616/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.714938:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.714939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.714941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829687.714945:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829687.714956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.714962:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78a8a80 x1874663333375744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.714974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333375744/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.714979:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.714981:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.714983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333375744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829687.714986:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333375872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829687.714996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.715002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78a9180 x1874663333375872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.715009:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333375872/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.715014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.715016:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.715018:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333375872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829687.715046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829687.715060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.715067:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78a8700 x1874663333376000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.715075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333376000/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.715085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.715087:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.715089:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829687.715093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829687.715103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829687.715108:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff9057c78a9500 x1874663333376128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829687.715116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333376128/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829687.715121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829687.715123:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829687.715125:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376128:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829688.098383:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829689.122406:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829690.146332:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829691.170424:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829692.194394:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829692.834293:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834307:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829692.834310:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834319:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834328:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834335:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829692.834373:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333376256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829692.834398:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834401:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829692.834403:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834406:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834410:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834413:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829692.834422:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829692.834428:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333376256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829692.834436:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333376384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829692.834445:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834448:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829692.834449:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834451:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834455:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834458:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829692.834462:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829692.834470:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333376512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829692.834477:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834479:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829692.834480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834484:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11503 00000800:00000100:2.0:1787829692.834488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829692.834489:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834496:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829692.834500:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829692.834508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333376640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829692.834515:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829692.834519:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834521:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834523:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834525:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829692.834528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829692.834534:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78ab100 x1874663333376256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829692.834538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333376768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829692.834569:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333376256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.834570:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834579:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829692.834583:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.834588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.834589:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834595:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.834598:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829692.834602:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834606:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333376256:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787829692.834608:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829692.834621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333376896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829692.834628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829692.834630:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376384:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829692.834632:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829692.834637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834640:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00080000:1.0:1787829692.834643:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834646:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829692.834649:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829692.834654:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829692.834660:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333377024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829692.834668:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78a9500 x1874663333376384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.834670:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829692.834678:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829692.834681:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829692.834687:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11503 00000100:00100000:2.0:1787829692.834689:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333376384/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.834692:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829692.834700:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829692.834702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829692.834704:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829692.834708:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834713:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829692.834716:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333377152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829692.834722:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829692.834743:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834751:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78a8700 x1874663333376512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834758:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333376512/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829692.834770:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829692.834781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78a9180 x1874663333376640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834794:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333376640/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834799:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834801:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834802:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829692.834806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829692.834815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834821:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78a8a80 x1874663333376768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333376768/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834833:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333376768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829692.834845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829692.834855:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78aa300 x1874663333376896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333376896/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834873:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834875:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333376896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829692.834880:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829692.834890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78a9c00 x1874663333377024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834902:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333377024/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829692.834913:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829692.834922:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829692.834927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff9057c78aa680 x1874663333377152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.834934:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333377152/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829692.834939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829692.834941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829692.834943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377152:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829693.218398:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829694.242360:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829695.266405:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829696.290408:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829697.314432:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829697.954318:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954329:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829697.954333:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829697.954344:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954354:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954362:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829697.954400:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333377280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829697.954426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954428:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829697.954430:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829697.954433:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954436:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954439:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954446:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829697.954457:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333377408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829697.954467:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954470:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829697.954472:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829697.954476:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333377280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829697.954478:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954484:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954486:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829697.954501:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333377536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829697.954508:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954511:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829697.954512:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829697.954515:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954518:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954520:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954523:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829697.954531:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333377664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829697.954536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829697.954544:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954547:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829697.954549:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829697.954552:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954558:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954562:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829697.954572:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333377792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829697.954576:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78abb80 x1874663333377280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.954585:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954591:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829697.954593:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829697.954598:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00100000:2.0:1787829697.954600:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333377280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.954601:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954610:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829697.954617:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954622:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829697.954625:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333377920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829697.954628:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333377280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829697.954648:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829697.954651:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829697.954652:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829697.954655:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377408:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829697.954659:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954665:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954670:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954676:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829697.954677:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.954688:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78a9f80 x1874663333377408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829697.954697:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333378048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829697.954702:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333377408/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.954729:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829697.954734:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829697.954737:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377408:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829697.954739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829697.954743:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829697.954751:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377536:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829697.954757:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11508 00000100:00080000:1.0:1787829697.954762:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829697.954767:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829697.954771:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829697.954776:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829697.954786:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333378176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829697.954794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78a9880 x1874663333377536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.954807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333377536/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.954815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.954820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829697.954824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829697.954840:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.954847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78a8000 x1874663333377664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.954854:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333377664/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.954859:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954861:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.954863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829697.954866:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829697.954886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.954896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78ab800 x1874663333377792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.954909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333377792/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.954914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.954918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333377792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829697.954921:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829697.954932:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.954946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78ab480 x1874663333377920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.954968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333377920/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.954973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.954975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.954977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333377920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829697.954990:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829697.955003:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.955010:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78aaa00 x1874663333378048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.955018:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333378048/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.955027:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.955029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.955031:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829697.955034:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829697.955044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829697.955049:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff9057c78a8380 x1874663333378176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.955056:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333378176/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.955061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829697.955063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829697.955065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378176:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829698.338395:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829699.362483:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829700.386497:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829701.410424:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829702.434347:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829703.074412:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.074427:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829703.074432:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829703.074448:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:1.0:1787829703.074463:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829703.074473:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829703.074536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333378304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.074571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.074575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829703.074577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829703.074583:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:1.0:1787829703.074587:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829703.074592:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829703.074604:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829703.074623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333378432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.074638:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.074642:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829703.074646:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829703.074652:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333378304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829703.074657:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:1.0:1787829703.074668:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829703.074676:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829703.074684:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829703.074702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333378560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.074713:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.074718:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829703.074721:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829703.074726:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11513 00000800:00000100:2.0:1787829703.074730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829703.074735:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829703.074745:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829703.074753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829703.074769:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333378688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.074793:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829703.074799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78aad80 x1874663333378304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.074800:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829703.074806:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829703.074827:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:1.0:1787829703.074835:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829703.074843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333378304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.074844:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829703.074864:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829703.074895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829703.074904:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333378816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829703.074908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829703.074943:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829703.074953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333378304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829703.074956:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829703.074972:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829703.074987:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:1.0:1787829703.074994:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829703.075001:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075007:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378432:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829703.075008:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829703.075038:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333378944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829703.075044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075060:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78a8380 x1874663333378432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.075063:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.075075:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829703.075077:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333378432/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.075079:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829703.075096:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.075097:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:2.0:1787829703.075108:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829703.075111:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829703.075112:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378432:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829703.075121:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075131:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378560:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829703.075132:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829703.075162:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829703.075163:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333379072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829703.075174:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78aaa00 x1874663333378560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075217:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333378560/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.075274:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829703.075289:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829703.075296:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829703.075297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.075317:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11513 00000100:00080000:2.0:1787829703.075322:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829703.075327:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829703.075328:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378560:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829703.075339:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075340:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378688:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1787829703.075355:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829703.075370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829703.075379:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333379200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829703.075383:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78ab480 x1874663333378688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075431:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333378688/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829703.075443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829703.075446:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829703.075449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829703.075456:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829703.075471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075479:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78ab800 x1874663333378816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075486:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333378816/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829703.075491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829703.075493:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829703.075495:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333378816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829703.075508:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829703.075519:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075525:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78a8000 x1874663333378944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075532:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333378944/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829703.075538:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829703.075540:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829703.075542:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333378944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829703.075545:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829703.075556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075561:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78a9880 x1874663333379072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075568:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333379072/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829703.075573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829703.075575:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829703.075577:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829703.075580:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829703.075590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829703.075595:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9057c78a9f80 x1874663333379200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829703.075602:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333379200/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829703.075607:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829703.075609:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829703.075611:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379200:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829703.458500:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829704.482354:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829705.506382:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829706.530436:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829707.554379:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829708.194329:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829708.194344:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194354:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829708.194420:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333379328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.194451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194454:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.194456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194476:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829708.194486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333379456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.194495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194500:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829708.194501:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333379328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829708.194502:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194511:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194514:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194517:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194520:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829708.194530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333379584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.194537:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194540:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.194541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194545:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194564:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194567:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194571:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829708.194579:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333379712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.194587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.194591:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194594:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194597:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194600:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194604:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829708.194613:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333379840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829708.194616:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379456:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829708.194631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.194636:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194640:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194652:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829708.194655:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379584:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829708.194661:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194668:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829708.194677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787829708.194685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333379968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829708.194698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379840:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829708.194700:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.194709:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194714:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00100000:2.0:1787829708.194718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379968:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829708.194720:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194728:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829708.194746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333380096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.194757:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.194760:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.194762:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.194765:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11518 00000100:00080000:3.0:1787829708.194768:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.194771:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829708.194774:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829708.194778:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829708.194786:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333380224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829708.194809:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380224:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829708.578461:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829709.602404:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829710.626385:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829711.266513:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829711.266550:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829711.266564:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266603:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266612:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266622:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266632:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266644:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829711.266656:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829711.266661:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb220380 x1874663333379840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829711.266670:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829711.266699:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333379840/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266713:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266718:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829711.266730:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb222d80 x1874663333379456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266743:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333379456/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266754:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829711.266764:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb223100 x1874663333379584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266776:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333379584/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266783:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266787:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829711.266798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb221880 x1874663333379712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266809:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333379712/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266823:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333379712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829711.266830:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb221c00 x1874663333379328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333379328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266854:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266874:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333379328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829711.266907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb222a00 x1874663333380224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266921:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333380224/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266929:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266933:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829711.266944:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb220700 x1874663333380096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333380096/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.266970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.266972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829711.266979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829711] req@ffff9057fb222300 x1874663333379968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829711.266991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333379968/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829711.266999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829711.267003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829711.267005:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333379968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1787829711.650396:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829712.674357:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829713.314331:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314341:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829713.314345:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314355:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314366:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314374:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829713.314404:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333380352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314427:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314430:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829713.314432:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314435:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314438:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314441:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829713.314448:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829713.314462:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333380480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314474:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314478:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829713.314480:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829713.314485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333380352:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829713.314487:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314494:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314498:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829713.314504:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829713.314520:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333380608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314535:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829713.314537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314541:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314555:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314558:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829713.314563:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829713.314574:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829713.314575:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333380736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314589:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314593:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829713.314595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314599:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314603:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314608:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829713.314613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829713.314623:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb221500 x1874663333380352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829713.314629:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333380864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314651:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829713.314658:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333380352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829713.314659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314669:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314674:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829713.314680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829713.314681:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829713.314691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787829713.314691:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829713.314700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333380352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829713.314707:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333380992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829713.314718:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829713.314722:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829713.314724:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314731:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00100000:2.0:1787829713.314732:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380480:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829713.314737:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314749:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829713.314755:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829713.314770:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829713.314787:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333381120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829713.314788:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb220e00 x1874663333380480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829713.314820:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829713.314825:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333380480/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829713.314826:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829713.314830:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829713.314838:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11524 00000100:00080000:3.0:1787829713.314859:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829713.314869:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829713.314871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829713.314875:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829713.314879:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.314886:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829713.314893:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333381248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829713.314897:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829713.314931:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.314946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb222680 x1874663333380608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.314957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333380608/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.314966:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.314969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.314973:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829713.314979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829713.315000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.315012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb221180 x1874663333380736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.315023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333380736/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.315031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.315034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.315038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829713.315044:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829713.315064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.315076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb223b80 x1874663333380864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.315088:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333380864/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.315096:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.315099:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.315102:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333380864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829713.315108:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829713.315129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.315141:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb220a80 x1874663333380992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.315152:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333380992/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.315160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.315163:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.315165:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333380992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829713.315186:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333381120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829713.315207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.315256:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb220000 x1874663333381120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.315271:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333381120/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.315279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.315282:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.315286:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333381120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829713.315292:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333381248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829713.315313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829713.315324:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829713/real 1787829713] req@ffff9057fb223800 x1874663333381248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829713.315335:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333381248/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829713.315342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829713.315345:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829713.315348:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333381248:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829713.698460:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829714.722380:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829715.746427:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829716.770411:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829717.794360:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829718.434376:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434387:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829718.434392:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434401:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434410:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434416:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829718.434449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333381376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434473:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434476:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829718.434477:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434480:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434486:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434493:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829718.434502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333381504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434509:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829718.434512:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434515:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434517:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434519:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434522:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829718.434530:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333381632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829718.434532:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333381376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829718.434539:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434552:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829718.434553:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434559:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434569:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829718.434577:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333381760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434583:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829718.434586:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434589:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434592:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434594:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829718.434602:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829718.434606:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333381888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829718.434617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434620:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434623:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434625:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434628:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829718.434636:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333382016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434642:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434644:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829718.434646:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434650:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00100000:2.0:1787829718.434651:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78abb80 x1874663333381376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829718.434652:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434659:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829718.434666:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829718.434677:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333382144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829718.434684:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829718.434687:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829718.434687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333381376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829718.434691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829718.434699:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11529 00000100:00080000:1.0:1787829718.434704:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829718.434708:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829718.434709:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829718.434711:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829718.434716:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829718.434720:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333382272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829718.434722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333381376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829718.434735:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829718.434758:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.434773:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78aa680 x1874663333381504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.434787:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333381504/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.434796:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.434800:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.434804:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829718.434848:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829718.434869:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.434878:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78a9c00 x1874663333381632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.434886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333381632/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.434896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.434899:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.434901:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829718.434905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829718.434917:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.434923:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78aa300 x1874663333381760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.434930:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333381760/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.434935:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.434937:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.434939:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829718.434942:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829718.434953:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.434959:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78a8a80 x1874663333381888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.434965:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333381888/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.434970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.434972:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.434975:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333381888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829718.434979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829718.434989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.434995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78a9180 x1874663333382016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.435002:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333382016/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.435009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.435012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.435016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829718.435022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829718.435041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.435054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78a8700 x1874663333382144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.435065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333382144/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.435071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.435073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.435075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829718.435079:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829718.435093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829718.435100:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829718/real 1787829718] req@ffff9057c78a9500 x1874663333382272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829718.435107:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333382272/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829718.435112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829718.435114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829718.435116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333382272:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829718.818363:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829719.842371:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829720.866495:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829721.890372:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829722.914405:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829723.554351:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554361:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829723.554366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554376:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554386:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554395:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829723.554432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333382400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829723.554464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554467:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829723.554468:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554477:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829723.554485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829723.554496:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333382528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829723.554507:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829723.554513:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554517:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554521:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554526:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829723.554531:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333382400:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829723.554534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829723.554549:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333382656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829723.554559:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829723.554573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554578:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554584:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554589:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829723.554592:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829723.554600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829723.554624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333382784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829723.554635:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829723.554639:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb223800 x1874663333382400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829723.554640:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829723.554644:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554654:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554661:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829723.554667:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333382400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829723.554668:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829723.554678:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829723.554681:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.554687:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.554691:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333382400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829723.554696:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333382912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829723.554701:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382528:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829723.554716:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554719:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829723.554721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554726:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11534 00000800:00000100:2.0:1787829723.554731:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829723.554732:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554738:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829723.554744:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb220000 x1874663333382528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829723.554744:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829723.554758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333383040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829723.554767:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333382528/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829723.554801:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829723.554802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829723.554808:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829723.554811:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829723.554812:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829723.554817:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382528:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829723.554819:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554826:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829723.554829:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829723.554834:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829723.554844:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333383168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829723.554847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382656:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829723.554861:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829723.554863:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829723.554865:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829723.554868:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11534 00000100:00080000:3.0:1787829723.554872:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829723.554874:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829723.554878:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829723.554883:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829723.554888:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb220a80 x1874663333382656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829723.554894:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333383296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829723.554906:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333382656/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.554914:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.554917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.554919:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829723.554924:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829723.554937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829723.554943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb223b80 x1874663333382784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829723.554950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333382784/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.554955:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.554957:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.554959:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829723.554963:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829723.554972:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829723.554978:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb221180 x1874663333382912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829723.554984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333382912/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.554989:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.554991:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.554993:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333382912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829723.554996:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829723.555006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829723.555012:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb222680 x1874663333383040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829723.555019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333383040/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.555024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.555026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.555027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829723.555037:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829723.555047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829723.555054:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb220e00 x1874663333383168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829723.555061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333383168/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.555066:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.555067:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.555069:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829723.555072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829723.555082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829723.555087:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829723/real 1787829723] req@ffff9057fb221500 x1874663333383296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829723.555094:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333383296/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829723.555099:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829723.555101:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829723.555103:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333383296:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829723.938382:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829724.962407:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829725.986373:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829727.010342:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829728.034481:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829728.674347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674358:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829728.674362:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674373:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674384:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674391:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829728.674435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333383424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829728.674464:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829728.674470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674475:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674479:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674484:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674494:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829728.674507:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333383424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829728.674511:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333383552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829728.674532:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674536:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829728.674538:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674542:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674553:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674557:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829728.674563:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829728.674582:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333383680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829728.674593:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674597:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829728.674599:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674603:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674608:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829728.674612:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb222300 x1874663333383424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829728.674615:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674623:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829728.674640:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333383808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829728.674647:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333383424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829728.674655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674663:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829728.674666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829728.674666:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829728.674674:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829728.674677:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00100000:2.0:1787829728.674683:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333383424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829728.674684:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674693:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674699:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829728.674711:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333383936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829728.674713:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829728.674753:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829728.674759:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829728.674761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829728.674765:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674772:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674777:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829728.674779:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb220700 x1874663333383552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829728.674785:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674796:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829728.674803:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333383552/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829728.674812:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333384064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829728.674821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829728.674825:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.674829:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829728.674830:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829728.674833:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829728.674838:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829728.674840:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674845:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674849:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829728.674854:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383680:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787829728.674856:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829728.674867:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333384192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829728.674874:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829728.674877:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829728.674878:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829728.674880:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11539 00000100:00080000:3.0:1787829728.674883:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829728.674887:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829728.674889:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829728.674891:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829728.674901:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333384320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829728.674909:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb222a00 x1874663333383680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.674926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333383680/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.674943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.674947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.674950:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829728.674957:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829728.674977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829728.674988:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb221c00 x1874663333383808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.675001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333383808/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.675009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.675013:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.675016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829728.675034:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829728.675053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829728.675063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb221880 x1874663333383936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.675076:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333383936/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.675085:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.675088:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.675092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333383936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829728.675098:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829728.675117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829728.675128:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb223100 x1874663333384064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.675140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333384064/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.675148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.675151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.675155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829728.675160:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829728.675177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829728.675186:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb222d80 x1874663333384192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.675196:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333384192/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.675204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.675207:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.675210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829728.675216:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829728.675263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829728.675285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff9057fb220380 x1874663333384320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829728.675298:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333384320/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829728.675307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829728.675310:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829728.675313:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333384320:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829729.058389:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829730.082426:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829731.106423:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829732.130475:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829733.154362:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829733.794285:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794292:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829733.794295:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829733.794304:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794313:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794321:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829733.794360:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333384448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829733.794386:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829733.794390:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829733.794393:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794396:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794399:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794406:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829733.794417:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333384576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829733.794425:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794427:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829733.794429:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829733.794432:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333384448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829733.794433:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794439:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794442:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794445:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829733.794453:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333384704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829733.794468:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829733.794474:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829733.794478:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794492:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794499:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829733.794500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794507:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829733.794524:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333384832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829733.794535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794538:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829733.794540:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829733.794548:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00100000:2.0:1787829733.794550:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb220380 x1874663333384448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829733.794553:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829733.794572:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333384448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829733.794581:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333384960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829733.794584:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794592:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.794595:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333384448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829733.794607:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829733.794610:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829733.794612:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829733.794615:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384576:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829733.794619:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794624:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794627:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794632:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829733.794640:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829733.794643:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333385088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829733.794653:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829733.794654:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb222d80 x1874663333384576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829733.794656:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829733.794659:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829733.794666:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794672:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829733.794676:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333384576/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829733.794677:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794683:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829733.794693:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333385216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829733.794694:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794715:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829733.794719:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829733.794723:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384576:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829733.794725:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829733.794730:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829733.794736:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384704:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829733.794739:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11544 00000100:00080000:3.0:1787829733.794747:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829733.794751:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829733.794758:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829733.794763:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.794778:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb223100 x1874663333384704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829733.794778:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333385344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829733.794811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333384704/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.794823:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794826:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.794828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829733.794835:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829733.794848:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.794855:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb221880 x1874663333384832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.794868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333384832/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.794876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794880:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.794884:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829733.794890:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829733.794910:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.794918:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb221c00 x1874663333384960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.794926:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333384960/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.794932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794934:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.794936:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333384960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829733.794949:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829733.794961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.794968:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb222a00 x1874663333385088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.794975:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333385088/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.794980:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.794982:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.794984:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829733.794987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829733.794996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.795002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb220700 x1874663333385216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.795029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333385216/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.795036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.795039:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.795041:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829733.795046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829733.795057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829733.795063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff9057fb222300 x1874663333385344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.795072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333385344/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.795078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829733.795080:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829733.795082:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333385344:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829734.178401:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829735.202423:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829736.226396:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829737.250414:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829738.274494:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829738.914373:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914388:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829738.914394:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914408:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914422:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914443:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829738.914492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333385472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914518:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914521:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829738.914524:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914528:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914531:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914534:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829738.914556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333385600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914564:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829738.914568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914571:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914573:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914576:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914579:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829738.914583:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333385472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829738.914589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333385728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914598:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914601:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829738.914602:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914605:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914606:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914609:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914613:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829738.914621:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333385856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914628:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914631:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829738.914632:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914635:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914637:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914640:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914642:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829738.914649:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333385984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914656:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829738.914677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914687:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11549 00000800:00000100:2.0:1787829738.914687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829738.914694:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914699:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914705:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829738.914721:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333386112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829738.914731:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914735:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829738.914737:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914742:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00100000:2.0:1787829738.914748:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78ab100 x1874663333385472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829738.914749:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914760:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914772:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829738.914782:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333385472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.914790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333386240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829738.914794:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.914802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.914809:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333385472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829738.914811:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829738.914817:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829738.914817:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385600:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829738.914822:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829738.914829:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11549 00000100:00080000:1.0:1787829738.914834:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829738.914840:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829738.914845:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829738.914849:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829738.914857:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78a9500 x1874663333385600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.914867:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333386368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829738.914870:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333385600/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.914899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.914902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.914905:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829738.914930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829738.914943:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.914949:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78a8700 x1874663333385728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.914957:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333385728/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.914967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.914970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.914972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829738.914975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829738.914986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.914992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78a9180 x1874663333385856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.914999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333385856/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.915004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.915006:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.915008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829738.915011:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829738.915021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.915027:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78a8a80 x1874663333385984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.915034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333385984/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.915039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.915041:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.915043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333385984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829738.915047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829738.915058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.915064:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78aa300 x1874663333386112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.915071:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333386112/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.915076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.915077:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.915079:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829738.915082:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829738.915092:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.915098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78a9c00 x1874663333386240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.915104:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333386240/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.915109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.915112:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.915114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829738.915117:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829738.915127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829738.915133:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff9057c78aa680 x1874663333386368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829738.915140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333386368/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829738.915146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829738.915147:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829738.915149:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333386368:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829739.298402:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829740.322451:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829741.346391:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829742.370529:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829743.394369:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829744.034355:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034366:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829744.034371:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034384:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034400:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034410:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829744.034461:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333386496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034491:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.034492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034496:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034499:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034503:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034513:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.034523:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333386624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034531:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034534:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.034535:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034538:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034541:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829744.034543:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333386496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829744.034547:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034553:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.034563:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333386752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034569:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034572:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.034573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034577:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034579:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034582:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034585:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829744.034593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333386880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034599:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034602:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.034603:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034605:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034610:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034612:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829744.034622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829744.034624:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333387008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034649:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034652:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.034654:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034658:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034667:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034671:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.034684:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333387136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.034691:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034694:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.034695:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034698:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:3.0:1787829744.034700:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829744.034703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb221500 x1874663333386496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829744.034706:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829744.034713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.034725:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333387264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829744.034732:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333386496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829744.034737:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.034745:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.034748:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.034753:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11554 00000100:00080000:2.0:1787829744.034755:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829744.034755:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.034762:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829744.034764:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1787829744.034766:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829744.034769:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333386496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829744.034783:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333387392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829744.034797:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829744.034819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.034828:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb220e00 x1874663333386624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.034836:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333386624/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.034843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.034852:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.034854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829744.034858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829744.034870:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.034877:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb222680 x1874663333386752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.034885:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333386752/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.034890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.034892:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.034893:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829744.034897:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829744.034909:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.034915:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb221180 x1874663333386880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.034922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333386880/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.034927:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.034928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.034931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333386880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829744.034935:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829744.034946:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.034951:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb223b80 x1874663333387008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.034958:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333387008/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.034964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.034965:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.034967:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829744.034970:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829744.034980:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.034986:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb220a80 x1874663333387136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.034993:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333387136/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.034998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.035000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.035002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829744.035005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829744.035015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.035021:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb220000 x1874663333387264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.035029:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333387264/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.035034:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.035036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.035038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829744.035041:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829744.035051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829744.035056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9057fb223800 x1874663333387392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829744.035063:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333387392/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829744.035068:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829744.035070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829744.035072:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333387392:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829744.418445:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829745.442439:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829746.466405:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829747.490517:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829748.514492:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829749.154323:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154352:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829749.154357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154371:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154387:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154413:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829749.154506:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333387520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154538:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154542:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829749.154544:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154550:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154554:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154558:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.154568:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829749.154581:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333387520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829749.154583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333387648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154613:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829749.154617:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154621:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154624:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154628:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.154650:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829749.154664:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333387776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154672:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154676:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829749.154677:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154680:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154684:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154687:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.154691:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829749.154698:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829749.154703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333387904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154713:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154716:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829749.154718:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154722:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154724:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154727:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.154730:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829749.154740:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333388032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154748:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829749.154750:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb223800 x1874663333387520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829749.154761:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829749.154766:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154795:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00080000:3.0:1787829749.154802:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829749.154806:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333387520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829749.154812:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829749.154823:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829749.154824:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829749.154829:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.154834:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333387520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829749.154844:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333388160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829749.154859:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829749.154863:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829749.154866:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829749.154871:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00100000:2.0:1787829749.154875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387648:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829749.154879:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154888:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.154895:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829749.154901:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829749.154915:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333388288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829749.154916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb220000 x1874663333387648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.154950:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333387648/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.154962:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829749.154962:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829749.154968:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829749.154969:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829749.154973:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829749.154974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387648:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829749.154982:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11559 00000100:00100000:2.0:1787829749.154985:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387776:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829749.154988:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829749.154995:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829749.155001:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829749.155006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829749.155018:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333388416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829749.155019:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb220a80 x1874663333387776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155064:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333387776/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155079:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155082:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829749.155088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829749.155102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829749.155108:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb223b80 x1874663333387904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333387904/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155122:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155124:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155126:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333387904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829749.155130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829749.155141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829749.155147:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb221180 x1874663333388032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155155:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333388032/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155162:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155165:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829749.155168:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829749.155179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829749.155185:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb222680 x1874663333388160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155192:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333388160/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155200:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155202:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829749.155215:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829749.155226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829749.155233:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb220e00 x1874663333388288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155241:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333388288/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155248:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155276:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829749.155281:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829749.155295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829749.155304:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9057fb221500 x1874663333388416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829749.155313:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333388416/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829749.155319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829749.155321:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829749.155323:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333388416:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829749.538570:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829750.562486:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829751.586507:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829752.610410:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829753.634466:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829754.274343:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274357:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829754.274360:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274369:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274377:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274383:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829754.274413:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333388544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274433:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274436:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829754.274437:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274440:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274444:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274451:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829754.274459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333388672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829754.274469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274471:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274473:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274478:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829754.274488:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333388800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274498:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274502:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829754.274504:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274509:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274513:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274517:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274524:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829754.274528:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333388544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829754.274541:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333388928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274571:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829754.274573:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274579:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274583:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274587:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829754.274603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333389056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274609:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829754.274613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274616:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274620:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274622:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274625:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829754.274634:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333389184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829754.274647:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829754.274651:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829754.274657:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274660:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274664:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274668:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274671:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829754.274695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333389312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829754.274706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829754.274709:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829754.274711:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829754.274718:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11565 00000100:00080000:1.0:1787829754.274724:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829754.274729:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829754.274737:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829754.274743:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78abb80 x1874663333388544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829754.274756:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333389440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829754.274775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333388544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.274805:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.274809:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.274813:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333388544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829754.274821:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829754.274843:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.274852:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78a9f80 x1874663333388672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.274861:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333388672/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.274867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.274870:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.274872:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829754.274876:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829754.274887:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.274893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78a9880 x1874663333388800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.274901:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333388800/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.274906:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.274908:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.274910:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829754.274935:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829754.274948:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.274955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78a8000 x1874663333388928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.274963:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333388928/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.274968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.274969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.274972:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333388928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829754.274975:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829754.274985:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.274991:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78ab800 x1874663333389056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.275005:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333389056/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.275010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.275012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.275015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829754.275019:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829754.275031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.275038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78ab480 x1874663333389184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.275046:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333389184/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.275051:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.275053:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.275054:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829754.275058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829754.275069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.275076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78aaa00 x1874663333389312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.275083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333389312/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.275088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.275090:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.275092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829754.275095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829754.275106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829754.275112:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829754/real 1787829754] req@ffff9057c78a8380 x1874663333389440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829754.275120:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333389440/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829754.275125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829754.275127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829754.275128:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333389440:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829754.658361:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829755.682435:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829756.706478:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829757.730424:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829758.754382:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829759.394364:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394376:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829759.394380:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394391:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394403:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394411:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829759.394456:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333389568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394483:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829759.394487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394491:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394493:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394496:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394505:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829759.394516:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333389696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829759.394547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394550:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394553:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394557:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829759.394565:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333389568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829759.394573:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333389824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394581:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394584:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829759.394586:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394588:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394591:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394593:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394596:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829759.394603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333389952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394610:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394612:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829759.394613:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394619:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394621:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394624:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829759.394631:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333390080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394637:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394639:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829759.394641:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394644:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394647:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394650:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394653:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829759.394659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829759.394663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333390208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394670:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829759.394675:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394678:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394680:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394683:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394686:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829759.394692:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333390336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829759.394698:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829759.394701:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829759.394702:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829759.394705:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11570 00000100:00080000:1.0:1787829759.394707:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829759.394710:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829759.394713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829759.394720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78aad80 x1874663333389568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829759.394722:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333390464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829759.394767:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333389568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.394780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.394786:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.394790:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333389568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829759.394799:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829759.394825:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.394840:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78a8380 x1874663333389696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.394854:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333389696/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.394864:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.394867:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.394871:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829759.394911:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829759.394935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.394948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78aaa00 x1874663333389824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.394961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333389824/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.394977:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.394981:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.394985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829759.394992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829759.395015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.395028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78ab480 x1874663333389952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.395040:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333389952/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.395048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.395051:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.395055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333389952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829759.395062:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829759.395082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.395093:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78ab800 x1874663333390080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.395105:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333390080/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.395114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.395117:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.395122:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829759.395128:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829759.395148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.395160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78a8000 x1874663333390208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.395172:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333390208/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.395180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.395183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.395186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829759.395192:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829759.395212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.395263:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78a9880 x1874663333390336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.395278:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333390336/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.395284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.395286:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.395288:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829759.395292:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829759.395310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829759.395318:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829759/real 1787829759] req@ffff9057c78a9f80 x1874663333390464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829759.395326:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333390464/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829759.395331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829759.395332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829759.395334:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333390464:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829759.778434:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829760.802369:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829761.826427:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829762.850352:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829763.874453:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829764.514324:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514334:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829764.514338:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514348:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11575 00000100:00080000:3.0:1787829764.514359:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514368:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829764.514408:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333390592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829764.514439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829764.514445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514451:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11575 00000100:00080000:3.0:1787829764.514455:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514461:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829764.514472:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829764.514480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333390592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829764.514502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333390720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829764.514517:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514521:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829764.514524:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514528:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11575 00000100:00080000:3.0:1787829764.514543:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514548:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829764.514553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829764.514558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829764.514576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333390848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829764.514588:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514592:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829764.514594:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829764.514598:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb222300 x1874663333390592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829764.514603:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11575 00000100:00080000:3.0:1787829764.514611:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514618:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829764.514622:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333390592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829764.514626:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829764.514633:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.514637:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.514641:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333390592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829764.514653:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333390976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829764.514663:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390720:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829764.514670:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514675:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829764.514678:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514684:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11575 00000800:00000100:2.0:1787829764.514687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829764.514692:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514699:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829764.514703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb220700 x1874663333390720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829764.514724:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829764.514734:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333390720/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829764.514742:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829764.514744:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333391104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829764.514748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.514763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829764.514777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829764.514782:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829764.514788:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829764.514791:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514799:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11575 00000800:00000100:2.0:1787829764.514800:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829764.514809:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829764.514816:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb222a00 x1874663333390848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829764.514818:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829764.514830:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829764.514834:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333390848/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829764.514843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.514848:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787829764.514849:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333391232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829764.514852:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829764.514870:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829764.514872:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390976:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829764.514879:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829764.514883:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514889:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11575 00000800:00000100:2.0:1787829764.514894:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829764.514894:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829764.514904:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829764.514909:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb221c00 x1874663333390976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829764.514913:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829764.514932:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333390976/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829764.514932:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333391360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829764.514965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.514971:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829764.514976:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829764.514977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333390976:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829764.514987:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829764.514988:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391104:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829764.514992:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829764.514999:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11575 00000100:00080000:3.0:1787829764.515005:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829764.515007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829764.515014:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829764.515020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb221880 x1874663333391104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829764.515023:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829764.515053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333391104/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829764.515061:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333391488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829764.515063:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.515070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.515076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829764.515088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829764.515109:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829764.515121:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb223100 x1874663333391232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829764.515134:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333391232/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829764.515144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.515147:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.515152:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829764.515169:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829764.515193:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829764.515206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb222d80 x1874663333391360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829764.515220:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333391360/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829764.515260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.515263:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.515266:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829764.515270:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829764.515294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829764.515303:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829764/real 1787829764] req@ffff9057fb220380 x1874663333391488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829764.515312:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333391488/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829764.515317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829764.515319:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829764.515321:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333391488:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829764.898368:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829765.922343:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829766.946410:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829767.970400:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829768.994423:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829769.634338:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634347:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829769.634351:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634359:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634367:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634373:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829769.634408:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333391616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634430:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634433:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829769.634434:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634437:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634439:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634442:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829769.634449:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829769.634473:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333391744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634481:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634484:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829769.634486:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634488:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634491:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634494:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829769.634498:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829769.634501:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333391616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829769.634508:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333391872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634516:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634518:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829769.634520:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634524:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634526:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829769.634529:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829769.634537:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333392000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634543:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634545:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829769.634547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634549:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634552:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634554:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829769.634558:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829769.634563:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829769.634569:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333392128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634577:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634580:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829769.634581:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634584:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634586:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634589:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829769.634592:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829769.634602:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb220380 x1874663333391616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829769.634603:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333392256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829769.634631:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829769.634642:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829769.634649:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634653:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634657:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634660:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333391616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829769.634664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829769.634670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634674:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634682:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333391616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829769.634683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333392384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829769.634690:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391744:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829769.634705:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829769.634708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829769.634711:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829769.634715:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829769.634716:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11580 00000100:00080000:3.0:1787829769.634724:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829769.634727:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634732:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb222d80 x1874663333391744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829769.634732:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829769.634749:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333392512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829769.634750:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333391744/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.634774:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634778:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634781:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.634803:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829769.634816:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634823:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb223100 x1874663333391872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.634830:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333391872/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.634840:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333391872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.634847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829769.634858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb221880 x1874663333392000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.634871:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333392000/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.634876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634879:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829769.634883:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829769.634893:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634907:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb221c00 x1874663333392128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.634915:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333392128/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.634919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634921:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634924:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829769.634927:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829769.634939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.634945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb222a00 x1874663333392256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.634953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333392256/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.634961:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.634964:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.634968:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.634981:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829769.635002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.635014:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb220700 x1874663333392384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.635022:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333392384/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.635028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.635030:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.635032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.635036:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829769.635049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829769.635056:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff9057fb222300 x1874663333392512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.635063:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333392512/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.635068:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.635070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829769.635072:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333392512:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829770.018389:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829771.042577:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829772.066451:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829773.090396:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829774.114343:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829774.754340:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754347:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829774.754350:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754357:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754365:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754370:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829774.754399:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333392640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754438:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754442:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829774.754443:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754448:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754451:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754454:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754468:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829774.754480:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333392640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829774.754484:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333392768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.754501:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754505:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754520:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754526:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829774.754538:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333392896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754547:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754549:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829774.754550:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754553:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754559:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754563:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829774.754572:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829774.754574:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333393024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754583:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.754586:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754590:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754593:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754595:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754600:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829774.754606:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829774.754614:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333393152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754626:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829774.754627:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393024:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787829774.754637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829774.754640:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754646:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00100000:1.0:1787829774.754649:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393152:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787829774.754652:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754658:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754663:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829774.754678:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333393280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754700:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754704:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.754705:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.754710:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754713:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754716:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754723:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829774.754725:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829774.754735:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333393408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.754743:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.754746:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.754747:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829774.754750:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393408:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829774.754753:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11585 00000100:00080000:2.0:1787829774.754758:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.754760:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829774.754764:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829774.754773:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333393536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829774.754794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393536:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829775.138382:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829776.162346:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829777.186424:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829777.826459:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829777.826499:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829777.826515:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826554:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826563:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826570:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826580:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826587:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829777.826596:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787829777.826605:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de1f80 x1874663333393280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829777.826606:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787829777.826636:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333393280/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826651:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826655:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829777.826662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de0e00 x1874663333393152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826673:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333393152/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826686:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826690:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826693:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829777.826699:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de0000 x1874663333393024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826709:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333393024/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826716:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826719:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826722:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829777.826727:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de1500 x1874663333392768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826738:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333392768/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826745:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826750:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829777.826758:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de1c00 x1874663333392896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826769:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333392896/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826776:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826782:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333392896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829777.826787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de0700 x1874663333392640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333392640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333392640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829777.826841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de2a00 x1874663333393536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826852:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333393536/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826860:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826863:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826866:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829777.826872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829777] req@ffff9057d0de3100 x1874663333393408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829777.826881:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333393408/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829777.826888:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829777.826891:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829777.826894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333393408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1787829778.210449:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829779.234419:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829779.874353:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874366:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829779.874371:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874391:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874404:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874414:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829779.874460:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333393664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874496:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874500:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829779.874502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874507:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874511:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874516:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829779.874541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333393792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874554:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829779.874556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874559:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874563:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874566:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829779.874583:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333393920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874593:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874597:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829779.874601:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874605:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874609:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874614:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874620:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829779.874628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333393664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829779.874639:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333394048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874651:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829779.874657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874663:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874667:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874672:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874677:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829779.874691:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333394176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874702:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874705:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829779.874707:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874712:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874715:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874719:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874723:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829779.874730:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829779.874736:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333394304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874745:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874748:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829779.874751:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874756:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00080000:1.0:1787829779.874760:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874764:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874776:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829779.874791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78abb80 x1874663333393664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.874793:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333394432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829779.874839:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829779.874844:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829779.874846:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829779.874853:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11590 00000100:00100000:2.0:1787829779.874857:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333393664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.874861:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829779.874873:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829779.874881:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829779.874882:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.874895:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829779.874903:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333394560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829779.874909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333393664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829779.874937:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829779.874977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78aa680 x1874663333393792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333393792/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875027:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829779.875033:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829779.875052:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875066:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78a9c00 x1874663333393920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333393920/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875092:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333393920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829779.875123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829779.875153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875164:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78aa300 x1874663333394048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875175:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333394048/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875182:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875186:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875188:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829779.875194:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829779.875211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875248:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78a8a80 x1874663333394176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875259:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333394176/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875270:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875273:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829779.875279:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829779.875300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875310:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78a9180 x1874663333394304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875321:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333394304/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875331:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875334:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829779.875340:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829779.875359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875369:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78a8700 x1874663333394432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875389:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333394432/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875400:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829779.875409:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829779.875427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829779.875437:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff9057c78a9500 x1874663333394560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829779.875449:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333394560/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829779.875457:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829779.875460:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829779.875462:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333394560:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829780.258436:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829781.282427:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829782.306373:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829783.330379:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829784.354434:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829784.994316:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994324:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829784.994328:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994337:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994356:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829784.994363:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829784.994394:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333394688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829784.994419:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994422:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829784.994424:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994428:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994432:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829784.994435:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994442:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829784.994454:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333394816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829784.994471:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994475:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829784.994478:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994482:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994485:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829784.994488:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994493:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829784.994512:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333394688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829784.994513:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333394944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829784.994535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994539:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829784.994541:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994546:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994549:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829784.994553:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994558:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829784.994571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333395072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829784.994580:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829784.994586:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994589:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994593:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829784.994599:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829784.994600:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829784.994623:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333395200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829784.994634:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994637:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829784.994639:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994644:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00080000:1.0:1787829784.994647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829784.994651:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994656:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829784.994662:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78ab100 x1874663333394688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829784.994674:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333395328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829784.994685:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333394688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.994700:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829784.994706:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829784.994707:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.994709:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.994714:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829784.994718:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00100000:2.0:1787829784.994725:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333394688:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829784.994728:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829784.994735:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394816:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829784.994736:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829784.994753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829784.994769:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333395456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829784.994773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829784.994792:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829784.994794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78a9500 x1874663333394816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.994807:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829784.994813:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829784.994824:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11595 00000100:00100000:2.0:1787829784.994824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333394816/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.994834:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.994843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.994848:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829784.994850:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.994859:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394816:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1787829784.994862:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829784.994882:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333395584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829784.994892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829784.994920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.994932:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78a8700 x1874663333394944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.994943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333394944/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.994949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.994951:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.994953:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333394944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829784.994956:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829784.994969:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.994975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78a9180 x1874663333395072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.994981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333395072/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.994985:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.994987:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.994989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829784.994992:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829784.995001:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.995006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78a8a80 x1874663333395200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.995012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333395200/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.995021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.995023:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.995025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829784.995028:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829784.995037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.995042:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78aa300 x1874663333395328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.995048:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333395328/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.995052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.995054:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.995055:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829784.995058:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829784.995068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.995073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78a9c00 x1874663333395456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.995078:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333395456/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.995082:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.995083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.995086:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829784.995097:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829784.995107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829784.995113:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff9057c78aa680 x1874663333395584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.995119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333395584/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829784.995124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829784.995125:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829784.995127:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333395584:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829785.378509:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829786.402351:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829787.426417:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829788.450496:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829789.474390:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829790.114281:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114289:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829790.114293:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114301:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114323:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114330:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829790.114361:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333395712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829790.114384:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114386:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829790.114388:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114391:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114393:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114395:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114404:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829790.114412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333395840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829790.114420:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333395712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829790.114423:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114427:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829790.114428:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114431:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114436:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114439:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829790.114447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333395968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829790.114454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114457:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829790.114458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114461:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114463:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114465:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114469:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829790.114476:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333396096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829790.114479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829790.114487:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114490:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829790.114492:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114496:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114500:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114503:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114506:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829790.114514:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333396224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829790.114522:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78abb80 x1874663333395712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829790.114523:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114531:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829790.114534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114541:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114545:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114549:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114549:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333395712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829790.114556:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829790.114567:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829790.114571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333396352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829790.114573:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114583:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333395712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829790.114585:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114593:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829790.114595:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114598:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114600:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114603:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114605:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829790.114612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333396480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829790.114619:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395840:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829790.114627:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829790.114630:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829790.114632:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829790.114635:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11600 00000100:00080000:1.0:1787829790.114640:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829790.114642:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829790.114646:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829790.114649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829790.114659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333396608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829790.114665:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78a9f80 x1874663333395840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114681:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333395840/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114688:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114691:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829790.114698:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829790.114711:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114717:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78a9880 x1874663333395968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114724:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333395968/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114729:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114731:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114733:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333395968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829790.114737:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829790.114747:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114763:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78a8000 x1874663333396096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333396096/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114782:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114784:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829790.114787:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829790.114799:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114804:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78ab800 x1874663333396224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114811:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333396224/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114818:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829790.114823:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829790.114834:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114839:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78ab480 x1874663333396352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333396352/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114859:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829790.114871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829790.114881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114886:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78aaa00 x1874663333396480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114894:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333396480/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114899:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829790.114905:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829790.114915:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829790.114920:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9057c78a8380 x1874663333396608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829790.114928:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333396608/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829790.114932:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829790.114934:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829790.114936:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333396608:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829790.498434:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829791.522380:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829792.546453:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829793.570398:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829794.594407:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829795.234369:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234379:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829795.234384:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234396:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234407:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829795.234415:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829795.234451:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333396736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829795.234475:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234478:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829795.234480:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234483:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234486:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829795.234495:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829795.234503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829795.234513:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333396736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829795.234515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333396864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829795.234528:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829795.234532:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234534:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829795.234540:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829795.234544:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829795.234552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333396992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829795.234570:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234574:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829795.234576:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234580:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234584:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829795.234590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829795.234594:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829795.234602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829795.234616:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333397120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829795.234624:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234627:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829795.234628:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234631:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234634:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829795.234637:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829795.234641:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829795.234642:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de0a80 x1874663333396736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829795.234654:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333397248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829795.234662:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333396736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829795.234665:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234673:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829795.234677:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829795.234680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829795.234681:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234687:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829795.234689:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829795.234693:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829795.234698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333396736:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1787829795.234707:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829795.234716:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333397376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829795.234721:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829795.234741:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234745:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829795.234748:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829795.234753:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234758:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829795.234763:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829795.234767:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829795.234769:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829795.234784:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de1180 x1874663333396864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829795.234794:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333397504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829795.234803:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333396864/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829795.234812:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829795.234819:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829795.234820:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829795.234822:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829795.234828:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787829795.234831:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11605 00000100:00080000:2.0:1787829795.234837:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829795.234838:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396864:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829795.234846:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829795.234852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396992:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1787829795.234852:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829795.234867:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333397632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829795.234884:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.234897:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de1880 x1874663333396992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.234907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333396992/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.234916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.234919:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.234923:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333396992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829795.234929:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829795.234951:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.234962:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de3800 x1874663333397120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.234975:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333397120/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.234981:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.234983:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.234985:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829795.234989:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829795.235001:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.235007:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de3100 x1874663333397248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.235014:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333397248/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.235020:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.235022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.235023:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829795.235035:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829795.235046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.235052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de2a00 x1874663333397376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.235059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333397376/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.235068:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.235072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.235075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829795.235081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829795.235099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.235110:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de0700 x1874663333397504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.235123:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333397504/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.235130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.235132:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.235134:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829795.235138:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829795.235153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829795.235160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829795/real 1787829795] req@ffff9057d0de1c00 x1874663333397632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829795.235167:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333397632/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829795.235172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829795.235173:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829795.235176:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333397632:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829795.618381:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829796.642401:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829797.666432:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829798.690411:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829799.714474:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829800.354343:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354351:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829800.354355:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354363:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354372:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354378:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829800.354408:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333397760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829800.354432:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354434:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829800.354436:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354439:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354442:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354446:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829800.354453:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829800.354466:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333397888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829800.354466:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333397760:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829800.354480:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354482:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829800.354483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354486:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354488:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354491:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829800.354495:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829800.354505:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333398016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829800.354511:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354513:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829800.354515:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354518:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354520:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354522:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829800.354525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829800.354529:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829800.354536:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333398144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829800.354544:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354546:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829800.354547:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354550:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354552:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354555:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829800.354559:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829800.354565:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333398272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829800.354571:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829800.354572:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78aad80 x1874663333397760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829800.354576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829800.354580:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354588:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354597:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354603:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354604:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333397760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829800.354607:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829800.354619:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333398400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829800.354622:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829800.354628:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354632:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829800.354633:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354635:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354637:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354641:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829800.354642:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829800.354644:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829800.354651:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333397760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829800.354654:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333398528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829800.354664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829800.354666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829800.354668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829800.354671:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11611 00000100:00080000:1.0:1787829800.354673:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829800.354675:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829800.354679:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829800.354682:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829800.354689:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333398656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829800.354709:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354725:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78a8380 x1874663333397888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354738:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333397888/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354747:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354755:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333397888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829800.354762:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829800.354783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78aaa00 x1874663333398016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354801:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333398016/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354806:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354808:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354811:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829800.354815:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829800.354829:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354835:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78ab480 x1874663333398144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354843:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333398144/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354848:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829800.354855:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829800.354875:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78ab800 x1874663333398272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333398272/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354895:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354897:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354898:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829800.354902:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829800.354918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354924:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78a8000 x1874663333398400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354931:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333398400/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354937:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354939:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354941:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829800.354944:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829800.354955:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.354960:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78a9880 x1874663333398528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.354967:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333398528/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.354973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.354975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.354977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829800.354990:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829800.355000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829800.355006:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829800/real 1787829800] req@ffff9057c78a9f80 x1874663333398656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829800.355013:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333398656/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829800.355018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829800.355020:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829800.355022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333398656:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829800.738390:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829801.762362:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829802.786389:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829803.810404:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829804.834378:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829805.474296:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474307:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829805.474311:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474321:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00080000:1.0:1787829805.474331:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474339:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829805.474376:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333398784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829805.474403:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829805.474410:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474416:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00080000:1.0:1787829805.474420:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474425:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829805.474434:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829805.474447:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333398784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829805.474452:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333398912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829805.474467:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829805.474475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00080000:1.0:1787829805.474485:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474490:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829805.474496:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829805.474506:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829805.474513:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333399040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829805.474526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829805.474533:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474538:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00080000:1.0:1787829805.474542:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474547:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829805.474551:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78abb80 x1874663333398784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829805.474564:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829805.474568:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333398784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.474577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.474583:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829805.474584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333399168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829805.474588:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333398784:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829805.474604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474607:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829805.474609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00100000:2.0:1787829805.474618:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398912:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829805.474621:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474630:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829805.474636:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829805.474661:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333399296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829805.474668:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829805.474681:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829805.474684:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829805.474686:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474690:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00100000:2.0:1787829805.474695:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78aa680 x1874663333398912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829805.474696:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474705:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829805.474714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829805.474722:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333398912/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829805.474730:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333399424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829805.474735:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829805.474749:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829805.474751:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829805.474753:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829805.474758:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333398912:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829805.474759:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829805.474769:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00100000:2.0:1787829805.474771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399040:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829805.474775:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829805.474781:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829805.474787:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829805.474795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829805.474801:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333399552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829805.474807:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78a9c00 x1874663333399040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829805.474824:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829805.474828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333399040/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829805.474830:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829805.474834:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829805.474842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829805.474845:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11616 00000100:00080000:2.0:1787829805.474849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829805.474852:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829805.474853:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399040:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829805.474860:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829805.474862:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399168:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1787829805.474867:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829805.474885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829805.474886:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333399680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829805.474895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78aa300 x1874663333399168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829805.474907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333399168/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.474913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.474916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829805.474918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829805.474930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829805.474939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829805.474945:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78a8a80 x1874663333399296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829805.474952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333399296/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.474957:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.474959:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829805.474961:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829805.474964:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829805.474973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829805.474979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78a9180 x1874663333399424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829805.474986:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333399424/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.474991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.474993:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829805.474995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829805.474998:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829805.475008:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829805.475013:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78a8700 x1874663333399552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829805.475019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333399552/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.475024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.475026:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829805.475028:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829805.475031:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829805.475040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829805.475046:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829805/real 1787829805] req@ffff9057c78a9500 x1874663333399680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829805.475052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333399680/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829805.475057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829805.475058:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829805.475061:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333399680:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829805.858417:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829806.882426:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829807.906433:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829808.930344:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829809.954460:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829810.594335:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594346:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829810.594359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594369:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594380:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594386:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829810.594420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333399808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.594445:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594447:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829810.594449:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594452:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594455:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594458:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594465:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829810.594474:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333399936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829810.594481:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333399808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829810.594484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594491:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.594494:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594497:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594500:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594503:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829810.594514:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333400064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.594520:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594523:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829810.594524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594527:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594529:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594531:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594535:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829810.594541:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333400192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.594547:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594550:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.594551:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594553:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594556:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594558:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594561:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829810.594565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829810.594572:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333400320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.594585:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594587:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829810.594588:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594591:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594594:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594597:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594600:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829810.594606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333400448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.594613:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829810.594618:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de1500 x1874663333399808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829810.594618:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.594621:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594627:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594631:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.594637:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594644:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829810.594652:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333400576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829810.594658:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333399808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829810.594661:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.594666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.594668:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.594675:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11621 00000100:00080000:2.0:1787829810.594680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829810.594683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829810.594683:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829810.594691:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829810.594692:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594698:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333399808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829810.594702:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333400704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829810.594724:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.594745:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594754:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de0000 x1874663333399936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594763:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333399936/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594768:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594771:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594774:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333399936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.594778:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.594788:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594794:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de0e00 x1874663333400064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333400064/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594807:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594808:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594810:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.594813:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829810.594824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594829:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de1f80 x1874663333400192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594836:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333400192/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594841:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594842:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594844:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829810.594848:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829810.594858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de1c00 x1874663333400320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594870:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333400320/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594879:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829810.594882:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.594891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de0700 x1874663333400448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594903:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333400448/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594908:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594911:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.594914:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.594924:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de2a00 x1874663333400576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333400576/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594948:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.594953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829810.594962:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829810.594968:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff9057d0de3100 x1874663333400704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.594974:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333400704/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.594979:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.594981:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829810.594982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333400704:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829810.978350:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829812.002488:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829813.026394:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829814.050546:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829815.074384:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829815.714592:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714603:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829815.714607:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714617:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714628:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714636:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829815.714670:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333400832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714694:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714697:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829815.714698:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714701:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714704:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714706:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714713:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829815.714722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333400960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714729:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714734:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829815.714736:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714740:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714744:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714747:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829815.714772:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333401088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829815.714777:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333400832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829815.714799:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714802:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829815.714803:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714806:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714809:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714812:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714817:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829815.714827:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333401216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714834:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714836:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829815.714838:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714841:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714843:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714845:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714848:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829815.714857:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333401344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714864:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714867:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829815.714868:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714871:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714874:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714876:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714881:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829815.714886:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829815.714893:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333401472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714902:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714906:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829815.714908:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714911:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714916:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829815.714918:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714921:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829815.714928:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333401600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829815.714935:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829815.714938:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829815.714941:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829815.714944:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11626 00000100:00080000:2.0:1787829815.714947:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829815.714953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de3800 x1874663333400832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829815.714953:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829815.714962:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829815.714979:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333401728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829815.714994:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333400832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715022:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333400832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829815.715031:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829815.715056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715068:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de1880 x1874663333400960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715080:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333400960/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715088:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715091:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715095:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333400960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.715149:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829815.715170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715181:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de1180 x1874663333401088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715192:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333401088/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715200:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715204:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.715213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829815.715271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715282:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de0a80 x1874663333401216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715293:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333401216/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715305:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715308:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829815.715314:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829815.715331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715341:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de3100 x1874663333401344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715351:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333401344/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715362:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715365:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829815.715371:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829815.715388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715397:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de2a00 x1874663333401472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715408:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333401472/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715420:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715423:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.715428:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829815.715445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715454:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de0700 x1874663333401600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715465:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333401600/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715476:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715479:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.715485:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829815.715501:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829815.715510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829815] req@ffff9057d0de1c00 x1874663333401728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829815.715521:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333401728/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829815.715536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829815.715539:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829815.715542:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333401728:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829816.098501:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829817.122332:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829818.146435:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829819.170367:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829820.194421:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829820.834332:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834340:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829820.834344:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834351:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834361:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834368:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829820.834404:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333401856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829820.834434:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834438:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829820.834440:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834447:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834452:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834457:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834462:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333401856:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1787829820.834467:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829820.834483:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333401984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829820.834493:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834498:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829820.834500:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834505:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834509:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829820.834517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829820.834518:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829820.834527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829820.834542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333402112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829820.834550:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834553:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829820.834556:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829820.834556:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78ab100 x1874663333401856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.834565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834572:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834577:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834581:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333401856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829820.834584:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829820.834596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829820.834599:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333402240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.834602:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.834610:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333401856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829820.834618:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834621:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829820.834623:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834627:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834629:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834631:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829820.834635:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829820.834636:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829820.834648:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333402368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829820.834657:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829820.834659:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829820.834660:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834663:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834667:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829820.834667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829820.834673:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829820.834678:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829820.834683:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78a9500 x1874663333401984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.834693:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333402496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829820.834696:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333401984/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.834725:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829820.834730:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829820.834731:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829820.834734:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333401984:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829820.834736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829820.834740:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829820.834743:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402112:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829820.834748:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11631 00000100:00080000:1.0:1787829820.834753:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834757:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829820.834762:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829820.834764:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834775:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78a8700 x1874663333402112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.834783:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333402624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829820.834788:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333402112/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.834815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.834820:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829820.834820:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.834824:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402112:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829820.834832:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829820.834833:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402240:192.168.203.156@tcp:8:kworker.0 00000100:00080000:1.0:1787829820.834838:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829820.834846:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11631 00000800:00000100:2.0:1787829820.834851:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829820.834853:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829820.834859:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834861:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78a9180 x1874663333402240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829820.834865:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829820.834876:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333402240/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.834883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829820.834884:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333402752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.834887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.834890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829820.834901:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829820.834918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78a8a80 x1874663333402368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829820.834934:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333402368/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.834940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829820.834942:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.834944:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333402368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829820.834948:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829820.834964:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829820.834975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78aa300 x1874663333402496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829820.834987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333402496/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.834995:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829820.834999:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.835002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829820.835015:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829820.835036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829820.835045:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78a9c00 x1874663333402624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829820.835053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333402624/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.835058:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829820.835060:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.835062:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829820.835073:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829820.835085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829820.835091:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff9057c78aa680 x1874663333402752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829820.835120:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333402752/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829820.835129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829820.835132:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829820.835135:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333402752:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829821.218512:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829822.242411:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829823.266376:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829824.290410:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829825.314367:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829825.954338:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954347:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829825.954350:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954359:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954369:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954376:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829825.954409:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333402880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829825.954436:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829825.954442:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954447:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954451:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954455:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954464:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829825.954475:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333402880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829825.954479:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333403008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829825.954496:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954499:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829825.954501:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954505:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954529:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954533:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954538:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829825.954540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829825.954554:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333403136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829825.954571:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829825.954575:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb221500 x1874663333402880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829825.954576:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829825.954579:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954583:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954587:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954595:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954600:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829825.954603:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333402880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.954612:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829825.954614:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333403264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829825.954618:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.954625:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333402880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829825.954634:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954638:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829825.954640:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954645:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00100000:2.0:1787829825.954648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403008:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829825.954650:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954658:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829825.954669:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954669:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829825.954680:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb220e00 x1874663333403008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829825.954685:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333403392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829825.954694:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333403008/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829825.954699:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954706:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829825.954713:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829825.954716:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.954720:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829825.954725:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00100000:2.0:1787829825.954728:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403008:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829825.954731:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829825.954739:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829825.954740:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403136:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787829825.954747:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829825.954762:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333403520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829825.954771:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829825.954779:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829825.954783:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829825.954785:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829825.954789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb222680 x1874663333403136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829825.954792:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954810:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829825.954814:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333403136/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829825.954819:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954828:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829825.954830:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.954836:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.954840:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829825.954843:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333403648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829825.954852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403264:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829825.954872:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829825.954876:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829825.954878:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829825.954882:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829825.954887:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11636 00000100:00080000:3.0:1787829825.954892:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829825.954894:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb221180 x1874663333403264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829825.954899:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829825.954907:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829825.954914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333403264/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829825.954923:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333403776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829825.954926:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.954936:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.954943:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829825.954961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829825.954979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829825.954989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb223b80 x1874663333403392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829825.955000:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333403392/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.955007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.955010:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.955013:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333403392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829825.955018:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829825.955032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829825.955042:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb220a80 x1874663333403520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829825.955053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333403520/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.955075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.955078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.955081:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829825.955087:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829825.955104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829825.955115:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb220000 x1874663333403648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829825.955126:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333403648/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.955133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.955135:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.955138:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829825.955152:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829825.955170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829825.955180:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff9057fb223800 x1874663333403776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829825.955191:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333403776/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.955205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829825.955207:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829825.955210:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333403776:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829826.338382:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829827.362379:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829828.386460:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829829.410441:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829830.434413:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829831.074381:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074390:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829831.074394:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074402:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074412:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074418:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829831.074449:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333403904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074481:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074484:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829831.074485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074489:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074492:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074495:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829831.074513:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333404032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074522:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074525:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829831.074527:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074531:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074534:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074541:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829831.074549:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333403904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829831.074551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333404160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074564:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074567:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829831.074569:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074573:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074576:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074579:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074583:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829831.074591:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333404288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074597:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074599:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829831.074601:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074603:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074606:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074607:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829831.074612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074612:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829831.074621:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333404416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074628:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074631:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829831.074632:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074635:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074637:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074639:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074642:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829831.074650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333404544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074656:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829831.074657:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de1f80 x1874663333403904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829831.074659:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829831.074662:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074667:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074670:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074674:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829831.074677:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829831.074681:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333404672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829831.074689:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829831.074693:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829831.074695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829831.074698:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11641 00000100:00080000:2.0:1787829831.074700:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829831.074704:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829831.074706:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333403904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829831.074707:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829831.074715:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333404800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829831.074726:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.074733:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.074738:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333403904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829831.074747:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829831.074770:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.074783:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de0e00 x1874663333404032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.074796:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333404032/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.074810:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.074813:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.074817:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829831.074838:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829831.074858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.074869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de0000 x1874663333404160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.074880:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333404160/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.074888:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.074890:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.074893:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829831.074899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829831.074919:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.074930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de1500 x1874663333404288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.074940:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333404288/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.074947:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.074950:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.074954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829831.074959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829831.074976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.074999:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de1c00 x1874663333404416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.075011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333404416/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.075019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.075022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.075038:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333404416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829831.075045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829831.075064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.075075:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de0700 x1874663333404544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.075086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333404544/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.075094:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.075097:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.075100:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829831.075106:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829831.075124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.075135:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de2a00 x1874663333404672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.075145:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333404672/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.075152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.075155:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.075158:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829831.075164:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829831.075181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829831.075191:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829831] req@ffff9057d0de3100 x1874663333404800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829831.075202:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333404800/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829831.075210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829831.075213:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829831.075216:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333404800:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829831.458462:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829832.482359:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829833.506378:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829834.530383:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829835.554475:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829836.194317:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194325:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829836.194328:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194335:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194343:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829836.194349:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829836.194379:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333404928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829836.194399:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194402:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829836.194403:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194406:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194408:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829836.194411:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194418:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829836.194427:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333404928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829836.194428:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333405056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829836.194440:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194443:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829836.194444:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194446:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194448:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829836.194450:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194453:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829836.194461:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333405184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829836.194467:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194469:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829836.194470:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194473:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194475:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829836.194479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829836.194481:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194486:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829836.194491:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333405312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829836.194497:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194499:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829836.194500:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194502:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194504:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829836.194507:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194509:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829836.194516:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333405440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829836.194521:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194523:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829836.194524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194527:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194531:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829836.194532:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78abb80 x1874663333404928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829836.194537:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829836.194555:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333404928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829836.194560:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333405568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829836.194564:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194571:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.194577:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333404928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829836.194578:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829836.194583:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829836.194586:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194590:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194594:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829836.194596:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405056:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829836.194600:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194606:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829836.194618:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333405696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829836.194620:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829836.194633:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829836.194637:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78a9f80 x1874663333405056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829836.194638:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829836.194642:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829836.194649:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11646 00000100:00080000:1.0:1787829836.194656:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829836.194660:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333405056/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829836.194662:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829836.194669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829836.194681:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194685:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829836.194689:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333405824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829836.194694:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.194707:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.194729:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.194740:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78a9880 x1874663333405184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.194751:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333405184/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.194758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.194764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.194770:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.194787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.194796:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78a8000 x1874663333405312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.194807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333405312/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.194814:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.194820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.194826:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.194842:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.194852:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78ab800 x1874663333405440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.194862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333405440/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.194870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.194875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333405440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.194880:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.194897:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.194908:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78ab480 x1874663333405568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.194918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333405568/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.194925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.194927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.194930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.194946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.194961:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.194970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78aaa00 x1874663333405696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.194981:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333405696/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.195001:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.195004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.195007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.195013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.195029:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829836.195039:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9057c78a8380 x1874663333405824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.195049:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333405824/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.195055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.195057:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829836.195060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333405824:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829836.578369:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829837.602390:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829838.626344:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829839.650373:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829840.674339:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829841.314288:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314295:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829841.314298:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314306:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314314:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314319:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829841.314349:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333405952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314370:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314372:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829841.314373:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314376:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314377:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314379:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314387:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829841.314394:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333406080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314401:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314403:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829841.314404:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314406:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314408:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314409:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314412:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829841.314418:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333406208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829841.314423:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333405952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829841.314426:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314432:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829841.314434:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314436:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314437:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314439:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314442:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829841.314449:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333406336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314454:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314456:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829841.314458:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314460:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314474:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314477:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829841.314486:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333406464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829841.314494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314502:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314504:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314507:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829841.314515:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829841.314519:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333406592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314529:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829841.314530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314533:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314535:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829841.314537:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314540:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829841.314548:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333406720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829841.314554:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829841.314556:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829841.314557:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829841.314559:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11652 00000100:00080000:3.0:1787829841.314561:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829841.314564:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406208:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829841.314566:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829841.314572:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829841.314583:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333406848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829841.314605:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829841.314629:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829841.314648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829841.314674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829841.314695:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406848:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829841.698379:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829842.722422:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829843.746330:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829844.386462:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829844.386502:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829844.386516:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386557:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386566:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386573:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386579:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386586:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386597:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829844.386605:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829844.386681:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb221500 x1874663333406848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386703:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333406848/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386716:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386720:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829844.386728:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb220a80 x1874663333406208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386739:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333406208/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386745:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386748:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386751:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829844.386759:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb223b80 x1874663333406336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386769:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333406336/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386775:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386778:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386780:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829844.386786:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb221180 x1874663333406464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333406464/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386805:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829844.386812:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb222680 x1874663333406592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386821:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333406592/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386830:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386832:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829844.386838:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb220e00 x1874663333406720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386848:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333406720/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386858:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386860:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333406720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829844.386866:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb223800 x1874663333405952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386877:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333405952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386884:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386887:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386890:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333405952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829844.386896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829841/real 1787829844] req@ffff9057fb220000 x1874663333406080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829844.386907:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333406080/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829844.386913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829844.386916:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829844.386919:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333406080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1787829844.770404:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829845.794396:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829846.434301:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434308:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829846.434311:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434318:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434326:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434331:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829846.434356:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333406976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829846.434376:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434379:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829846.434380:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434382:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434384:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434386:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829846.434392:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829846.434400:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333407104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829846.434407:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434409:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829846.434410:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434412:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434413:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434415:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829846.434418:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829846.434423:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333407232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829846.434431:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333406976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829846.434446:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434449:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829846.434451:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434456:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434459:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434463:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829846.434469:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829846.434480:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333407360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829846.434487:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434490:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829846.434491:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829846.434497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829846.434499:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434505:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434509:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829846.434514:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829846.434528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333407488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829846.434538:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829846.434539:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de0a80 x1874663333406976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829846.434544:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829846.434546:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434551:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434557:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434563:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829846.434574:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333406976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829846.434579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829846.434591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.434595:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829846.434602:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333407616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829846.434603:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333406976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829846.434617:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407104:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829846.434618:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829846.434627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434631:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00080000:2.0:1787829846.434635:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434640:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829846.434645:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829846.434649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829846.434662:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333407744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829846.434668:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de1180 x1874663333407104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829846.434675:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829846.434682:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829846.434689:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829846.434696:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11657 00000100:00100000:1.0:1787829846.434697:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333407104/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829846.434701:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829846.434709:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829846.434713:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829846.434716:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829846.434720:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.434726:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829846.434731:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333407872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829846.434754:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829846.434776:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.434787:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de1880 x1874663333407232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.434798:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333407232/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.434811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.434814:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.434817:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829846.434823:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829846.434841:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.434851:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de3800 x1874663333407360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.434862:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333407360/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.434869:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.434873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.434876:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829846.434882:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829846.434900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.434910:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de3100 x1874663333407488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.434936:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333407488/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.434944:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.434947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.434950:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333407488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829846.434961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829846.434981:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.434992:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de2a00 x1874663333407616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.435003:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333407616/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.435011:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.435014:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.435016:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829846.435022:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829846.435041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.435051:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de0700 x1874663333407744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.435061:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333407744/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.435069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.435072:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.435075:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829846.435081:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829846.435097:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829846.435108:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829846/real 1787829846] req@ffff9057d0de1c00 x1874663333407872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829846.435119:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333407872/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829846.435127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829846.435129:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829846.435132:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333407872:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829846.818351:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829847.842364:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829848.866430:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829849.890426:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829850.914466:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829851.554353:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554363:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829851.554367:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554378:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554391:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554399:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829851.554443:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333408000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554471:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554474:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829851.554475:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554479:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554481:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554484:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554493:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829851.554502:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333408128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554521:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829851.554523:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554527:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554529:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554532:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554536:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829851.554548:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333408000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829851.554548:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333408256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554573:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554575:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829851.554577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554580:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554582:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554585:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554589:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829851.554598:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333408384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554604:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554607:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829851.554609:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554612:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554615:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554618:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554621:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829851.554630:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333408512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554638:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554641:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829851.554645:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829851.554650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829851.554656:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554664:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554669:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829851.554683:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333408640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829851.554695:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554700:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829851.554703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de1500 x1874663333408000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829851.554704:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554720:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554729:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554736:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829851.554737:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333408000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829851.554751:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829851.554756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.554762:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829851.554766:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333408768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829851.554767:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333408000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829851.554793:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408128:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829851.554796:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829851.554801:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829851.554804:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829851.554808:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11662 00000100:00080000:2.0:1787829851.554815:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829851.554820:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829851.554823:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829851.554830:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829851.554842:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333408896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829851.554849:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de0000 x1874663333408128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.554869:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333408128/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.554880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.554900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.554905:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829851.554933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829851.554957:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.554970:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de0e00 x1874663333408256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.554984:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333408256/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.554992:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.554995:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.554997:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829851.555002:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829851.555018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.555026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de1f80 x1874663333408384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.555033:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333408384/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.555038:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.555040:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.555043:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829851.555047:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829851.555058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.555063:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de1c00 x1874663333408512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.555070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333408512/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.555076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.555078:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.555080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333408512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829851.555083:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829851.555094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.555100:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de0700 x1874663333408640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.555107:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333408640/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.555112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.555114:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.555116:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829851.555120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829851.555135:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.555147:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de2a00 x1874663333408768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.555159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333408768/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.555168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.555172:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.555175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829851.555180:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829851.555196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829851.555203:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829851/real 1787829851] req@ffff9057d0de3100 x1874663333408896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829851.555211:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333408896/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829851.555216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829851.555219:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829851.555254:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333408896:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829851.938378:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829852.962418:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829853.986318:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829855.010387:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829856.034390:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829856.674352:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674362:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829856.674365:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674373:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674382:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674388:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829856.674420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333409024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829856.674444:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674446:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829856.674447:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674450:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674452:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674455:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674462:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829856.674470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333409152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829856.674478:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674482:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829856.674485:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829856.674486:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333409024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829856.674488:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674493:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674497:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674500:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829856.674508:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333409280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829856.674514:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674516:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829856.674518:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674520:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674522:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674524:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829856.674535:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333409408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829856.674541:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674543:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829856.674544:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674547:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674550:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674552:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674556:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829856.674560:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829856.674570:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333409536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829856.674581:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674584:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829856.674587:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674592:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00100000:1.0:1787829856.674596:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de3800 x1874663333409024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829856.674598:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674608:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829856.674615:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333409024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829856.674615:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829856.674629:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.674634:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829856.674634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333409664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829856.674643:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333409024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829856.674648:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674652:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829856.674653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674657:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674663:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674666:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829856.674672:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333409792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829856.674683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409152:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829856.674683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829856.674692:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829856.674695:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829856.674700:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11667 00000100:00080000:2.0:1787829856.674704:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829856.674709:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829856.674715:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829856.674719:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829856.674731:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333409920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829856.674739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de1880 x1874663333409152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.674751:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333409152/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.674757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.674760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.674762:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.674768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829856.674788:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.674795:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de1180 x1874663333409280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.674802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333409280/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.674808:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.674810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.674812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.674816:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829856.674827:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.674836:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de0a80 x1874663333409408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.674860:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333409408/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.674870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.674873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.674877:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829856.674884:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829856.674904:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.674916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de3100 x1874663333409536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.674929:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333409536/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.674938:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.674941:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.674945:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333409536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829856.674952:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829856.674973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.674984:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de2a00 x1874663333409664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.674997:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333409664/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.675002:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.675004:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.675007:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.675020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829856.675034:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.675041:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de0700 x1874663333409792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.675048:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333409792/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.675053:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.675055:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.675056:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.675060:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829856.675069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829856.675075:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff9057d0de1c00 x1874663333409920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.675081:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333409920/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.675100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.675102:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829856.675104:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333409920:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829857.058469:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829858.082388:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829859.106413:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829860.130348:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829861.154354:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829861.794311:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829861.794321:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829861.794326:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829861.794335:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794346:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829861.794354:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829861.794389:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333410048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829861.794418:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829861.794422:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829861.794424:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829861.794431:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794436:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829861.794447:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333410048:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829861.794452:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829861.794464:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829861.794482:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333410176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829861.794500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829861.794505:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829861.794508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829861.794512:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829861.794518:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794523:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829861.794528:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829861.794535:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829861.794541:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78aad80 x1874663333410048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829861.794552:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333410304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829861.794558:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333410048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794570:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.794573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.794578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829861.794579:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829861.794584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829861.794585:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333410048:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829861.794595:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794600:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829861.794604:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829861.794610:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829861.794618:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829861.794625:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333410432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829861.794638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829861.794640:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829861.794646:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829861.794648:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829861.794654:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78a8380 x1874663333410176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794656:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794663:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829861.794669:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333410176/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794671:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829861.794681:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829861.794683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.794688:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829861.794691:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829861.794699:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829861.794700:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333410560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829861.794712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829861.794716:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829861.794721:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829861.794723:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829861.794726:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78aaa00 x1874663333410304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794732:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794741:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829861.794741:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333410304/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794750:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829861.794751:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.794759:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829861.794760:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829861.794761:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829861.794776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829861.794780:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333410688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829861.794792:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829861.794795:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829861.794800:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829861.794803:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829861.794807:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78ab480 x1874663333410432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794825:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794834:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829861.794839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333410432/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794844:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829861.794853:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829861.794855:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829861.794857:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829861.794863:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829861.794868:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829861.794875:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333410816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829861.794884:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829861.794892:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829861.794896:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78ab800 x1874663333410560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794899:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829861.794904:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829861.794910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333410560/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.794914:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11672 00000100:00080000:1.0:1787829861.794923:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.794924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.794930:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829861.794934:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829861.794935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333410560:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1787829861.794946:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829861.794950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829861.794966:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333410944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829861.794967:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829861.794977:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78a8000 x1874663333410688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829861.794988:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333410688/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829861.794994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.794996:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829861.794998:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829861.795003:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829861.795014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829861.795020:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78a9880 x1874663333410816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829861.795028:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333410816/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829861.795032:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.795034:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829861.795036:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829861.795043:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829861.795053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829861.795058:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff9057c78a9f80 x1874663333410944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829861.795065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333410944/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829861.795070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829861.795071:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829861.795073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333410944:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829862.178352:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829863.202445:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829864.226410:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829865.250415:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829866.274420:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829866.914317:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914325:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829866.914329:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914337:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914347:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914353:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829866.914385:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333411072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914408:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914411:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829866.914412:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914415:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914417:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914420:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914427:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829866.914435:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333411200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914449:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914451:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829866.914453:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914456:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914459:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914463:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914468:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829866.914480:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333411328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829866.914492:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333411072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829866.914497:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829866.914510:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914515:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914519:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914524:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914530:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829866.914544:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333411456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914551:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914553:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829866.914555:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914558:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914564:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914567:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829866.914574:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333411584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914580:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829866.914584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914587:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914589:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914591:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914594:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829866.914600:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829866.914606:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333411712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914614:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829866.914616:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829866.914619:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914623:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914627:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829866.914632:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829866.914641:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333411840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829866.914655:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829866.914658:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78abb80 x1874663333411072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.914659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829866.914665:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829866.914684:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11677 00000100:00080000:1.0:1787829866.914688:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829866.914690:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333411072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.914702:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829866.914714:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829866.914718:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.914727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829866.914728:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333411968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829866.914742:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333411072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829866.914757:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829866.914802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.914818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78aa680 x1874663333411200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.914831:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333411200/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.914837:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.914840:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.914842:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829866.914867:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829866.914883:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.914891:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78a9c00 x1874663333411328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.914899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333411328/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.914904:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.914907:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.914909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829866.914914:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829866.914926:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.914933:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78aa300 x1874663333411456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.914940:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333411456/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.914946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.914947:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.914949:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829866.914953:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829866.914965:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.914971:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78a8a80 x1874663333411584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.914979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333411584/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.914984:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.914986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.914988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333411584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829866.914991:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829866.915002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.915009:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78a9180 x1874663333411712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.915016:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333411712/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.915022:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.915024:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.915025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829866.915029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829866.915040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.915045:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78a8700 x1874663333411840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.915052:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333411840/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.915057:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.915059:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.915062:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829866.915065:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829866.915075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829866.915081:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff9057c78a9500 x1874663333411968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829866.915088:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333411968/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829866.915098:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829866.915100:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829866.915102:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333411968:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829867.298420:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829868.322374:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829869.346406:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829870.370418:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829871.394452:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829872.034326:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034339:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829872.034344:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.034357:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034382:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829872.034432:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333412096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034465:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034469:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829872.034471:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.034477:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034481:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034485:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829872.034497:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829872.034515:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333412224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829872.034521:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333412096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829872.034535:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034539:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829872.034541:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.034546:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034551:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034555:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829872.034561:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829872.034576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333412352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034587:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034590:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829872.034592:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.034597:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034601:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034605:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829872.034611:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829872.034613:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829872.034627:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333412480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034642:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829872.034645:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.034650:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034654:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034659:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829872.034665:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829872.034674:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78ab100 x1874663333412096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.034679:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333412608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034704:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829872.034710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829872.034713:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333412096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.034718:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034736:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829872.034756:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.034758:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829872.034767:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829872.034772:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829872.034775:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333412096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829872.034790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333412736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034802:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.034806:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829872.034809:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829872.034811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412224:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829872.034817:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034823:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829872.034828:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829872.034834:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829872.034847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829872.034852:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333412864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829872.034868:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829872.034869:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78a9500 x1874663333412224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.034886:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829872.034889:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829872.034891:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333412224/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.034897:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11682 00000100:00080000:1.0:1787829872.034907:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829872.034910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.034915:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829872.034920:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:1.0:1787829872.034926:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829872.034927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829872.034940:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829872.034946:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333412992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829872.034971:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.034988:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78a8700 x1874663333412352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035004:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333412352/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035014:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035017:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829872.035028:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829872.035048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.035059:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78a9180 x1874663333412480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035070:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333412480/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035079:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035086:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829872.035093:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829872.035115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.035128:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78a8a80 x1874663333412608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333412608/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035152:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333412608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829872.035162:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829872.035181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.035191:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78aa300 x1874663333412736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035204:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333412736/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035216:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035246:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829872.035269:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829872.035292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.035305:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78a9c00 x1874663333412864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035324:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333412864/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035337:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035341:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829872.035347:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829872.035368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829872.035381:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9057c78aa680 x1874663333412992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829872.035394:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333412992/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829872.035403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829872.035406:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829872.035410:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333412992:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829872.418392:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829873.442399:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829874.466354:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829875.490390:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829876.514421:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829877.154334:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154343:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829877.154346:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154354:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154362:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154368:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829877.154401:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333413120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154423:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154426:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829877.154427:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154430:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154433:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154436:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154443:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829877.154451:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333413248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154461:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154465:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829877.154467:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154472:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154476:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829877.154482:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333413120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829877.154483:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154489:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829877.154502:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333413376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154513:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154516:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829877.154518:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154522:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154526:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154529:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154534:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829877.154545:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333413504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154553:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154557:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829877.154559:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154563:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11687 00000800:00000100:2.0:1787829877.154569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829877.154571:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154578:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154584:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829877.154596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333413632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154604:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829877.154608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154611:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154614:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154619:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829877.154620:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb222300 x1874663333413120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829877.154624:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829877.154637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333413760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829877.154644:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154648:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829877.154650:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333413120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829877.154650:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154658:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154662:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154667:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154675:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829877.154679:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829877.154683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333413888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829877.154689:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829877.154694:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829877.154699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829877.154701:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333413120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829877.154701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829877.154720:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11687 00000100:00080000:3.0:1787829877.154726:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829877.154730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829877.154734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829877.154735:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829877.154745:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333414016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829877.154769:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.154790:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb220700 x1874663333413248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.154804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333413248/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.154813:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.154817:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.154820:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.154845:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.154867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.154879:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb222a00 x1874663333413376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.154892:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333413376/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.154900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.154904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.154908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.154915:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.154936:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.154948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb221c00 x1874663333413504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.154961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333413504/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.154970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.154974:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.154977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829877.154984:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.155006:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.155019:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb221880 x1874663333413632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.155032:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333413632/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.155041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.155044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.155048:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333413632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829877.155055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333413760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.155075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.155088:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb223100 x1874663333413760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.155101:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333413760/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.155110:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.155113:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.155117:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333413760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.155123:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333413888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.155143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.155153:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb222d80 x1874663333413888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.155165:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333413888/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.155174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.155177:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.155181:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333413888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.155188:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.155209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829877.155247:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9057fb220380 x1874663333414016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.155261:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333414016/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.155277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.155280:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829877.155284:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414016:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829877.538417:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829878.562447:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829879.586362:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829880.610400:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829881.634391:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829882.274347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274357:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829882.274362:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274372:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274382:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274390:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829882.274428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333414144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274452:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829882.274456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274459:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274463:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274466:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829882.274483:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333414272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274491:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829882.274494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274497:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274499:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274502:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829882.274508:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333414144:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829882.274508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829882.274529:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333414400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274541:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274545:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829882.274547:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274552:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274566:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274571:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829882.274577:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274580:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829882.274593:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333414528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274605:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274608:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829882.274609:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274613:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274617:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274619:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274622:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829882.274625:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb220380 x1874663333414144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829882.274633:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333414656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274651:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274655:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829882.274656:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829882.274661:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333414144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829882.274661:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274663:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829882.274680:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274685:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829882.274697:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829882.274699:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333414784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829882.274708:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829882.274712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829882.274717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333414144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829882.274717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829882.274721:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274729:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274732:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829882.274735:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414272:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829882.274738:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829882.274754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333414912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829882.274761:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829882.274764:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829882.274765:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829882.274768:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11693 00000100:00080000:3.0:1787829882.274772:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829882.274773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829882.274777:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829882.274782:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829882.274798:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333415040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829882.274799:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb222d80 x1874663333414272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.274830:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333414272/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.274839:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.274844:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.274848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829882.274871:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829882.274890:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.274902:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb223100 x1874663333414400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.274914:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333414400/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.274923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.274927:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.274930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829882.274937:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829882.274956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.274967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb221880 x1874663333414528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.274979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333414528/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.274988:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.274991:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.274995:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829882.275001:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829882.275021:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.275032:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb221c00 x1874663333414656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.275044:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333414656/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.275053:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.275056:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.275060:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333414656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829882.275066:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829882.275084:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.275095:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb222a00 x1874663333414784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.275108:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333414784/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.275116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.275119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.275123:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829882.275130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829882.275149:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.275160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb220700 x1874663333414912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.275178:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333414912/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.275185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.275189:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.275192:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333414912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829882.275198:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829882.275242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829882.275266:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829882/real 1787829882] req@ffff9057fb222300 x1874663333415040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829882.275278:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333415040/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829882.275286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829882.275289:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829882.275292:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415040:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829882.658385:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829883.682399:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829884.706364:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829885.730444:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829886.754448:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829887.394385:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394395:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829887.394400:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394411:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394422:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394429:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1787829887.394468:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158c000 x1874663333415168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394499:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394502:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829887.394504:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394509:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394512:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394516:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829887.394525:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829887.394540:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f800 x1874663333415296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394551:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394554:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829887.394557:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829887.394559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333415168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1787829887.394563:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394571:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394575:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829887.394579:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829887.394592:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e680 x1874663333415424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394602:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394606:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829887.394608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394612:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394616:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394620:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829887.394625:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829887.394631:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1787829887.394655:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158df80 x1874663333415552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394666:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394670:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829887.394672:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394686:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394691:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394695:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829887.394701:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158c000 x1874663333415168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829887.394702:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829887.394716:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158ca80 x1874663333415680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394724:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394729:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829887.394729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158c000 x1874663333415168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829887.394732:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394738:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394744:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394751:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829887.394753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829887.394757:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829887.394761:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.394768:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333415168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1787829887.394783:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158f100 x1874663333415808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829887.394795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415296:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1787829887.394797:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394805:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829887.394807:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829887.394812:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394825:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829887.394827:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1787829887.394831:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1787829887.394837:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829887.394847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158f800 x1874663333415296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829887.394851:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158e300 x1874663333415936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829887.394884:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829887.394888:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829887.394891:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829887.394892:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f800 x1874663333415296/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829887.394895:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11698 00000100:00080000:0.0:1787829887.394901:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829887.394910:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829887.394911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829887.394916:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829887.394918:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.394925:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1787829887.394933:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057f158dc00 x1874663333416064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829887.394936:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829887.394960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.394975:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158e680 x1874663333415424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.394987:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e680 x1874663333415424/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.394996:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.394999:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829887.395010:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829887.395028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.395038:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158df80 x1874663333415552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.395051:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158df80 x1874663333415552/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.395059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.395063:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395066:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829887.395072:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829887.395089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.395098:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158ca80 x1874663333415680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.395109:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158ca80 x1874663333415680/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.395116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.395119:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395123:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333415680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829887.395140:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829887.395157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.395169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158f100 x1874663333415808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.395181:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158f100 x1874663333415808/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.395189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.395192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395195:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829887.395201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829887.395217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.395252:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158e300 x1874663333415936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.395265:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158e300 x1874663333415936/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.395274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.395277:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395281:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333415936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829887.395287:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829887.395320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829887.395332:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829887/real 1787829887] req@ffff9057f158dc00 x1874663333416064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829887.395343:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057f158dc00 x1874663333416064/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829887.395357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829887.395360:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829887.395363:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057f158dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416064:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829887.778428:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829888.802428:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829889.826395:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829890.850415:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829891.874405:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829892.514367:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514377:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829892.514381:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514389:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514398:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514404:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829892.514440:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333416192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514475:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514478:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829892.514480:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514483:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514492:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514506:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829892.514518:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333416320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514527:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514530:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829892.514532:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514535:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514537:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514539:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829892.514549:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333416448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514557:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514559:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829892.514561:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514568:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514570:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829892.514573:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333416192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829892.514575:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514587:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829892.514613:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333416576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514620:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514623:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829892.514624:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514626:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514630:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514632:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514637:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829892.514647:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333416704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514654:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514656:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829892.514657:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514660:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514664:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514667:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514671:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829892.514677:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333416832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514683:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514687:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829892.514688:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514690:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514692:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514694:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514697:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829892.514705:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829892.514706:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333416960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829892.514719:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829892.514721:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829892.514723:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829892.514726:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11703 00000100:00080000:2.0:1787829892.514728:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829892.514730:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829892.514732:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829892.514738:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333417088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829892.514793:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de1f80 x1874663333416192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.514817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333416192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.514831:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.514837:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.514842:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333416192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829892.514852:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829892.514881:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.514895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de0e00 x1874663333416320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.514910:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333416320/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.514919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.514923:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.514927:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829892.514958:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829892.514976:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.514984:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de0000 x1874663333416448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.514992:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333416448/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.514998:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515003:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829892.515007:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829892.515019:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.515025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de1500 x1874663333416576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.515038:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333416576/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.515043:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515045:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829892.515051:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829892.515063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.515068:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de1c00 x1874663333416704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.515075:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333416704/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.515080:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515083:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515085:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333416704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829892.515088:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829892.515099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.515105:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de0700 x1874663333416832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.515112:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333416832/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.515122:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515125:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515127:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829892.515130:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829892.515141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.515146:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de2a00 x1874663333416960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.515153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333416960/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.515157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515160:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515162:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333416960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829892.515166:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829892.515176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829892.515182:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829892/real 1787829892] req@ffff9057d0de3100 x1874663333417088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829892.515189:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333417088/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829892.515195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829892.515197:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829892.515199:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417088:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829892.898525:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829893.922468:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829894.946363:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829895.970496:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829896.994425:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829897.634385:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634398:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829897.634404:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634419:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634432:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634442:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829897.634492:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333417216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634521:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634525:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829897.634527:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634530:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634534:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634537:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634546:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829897.634556:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333417344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634578:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634583:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829897.634584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333417216:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829897.634585:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634590:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634597:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634602:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634607:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829897.634627:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333417472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634638:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634642:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829897.634644:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634648:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634653:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634657:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829897.634664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634664:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829897.634683:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333417600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634695:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634699:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829897.634701:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634707:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00100000:1.0:1787829897.634710:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb220000 x1874663333417216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829897.634714:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634723:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634729:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829897.634739:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333417216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829897.634743:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333417728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634765:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829897.634775:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829897.634780:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829897.634780:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634785:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829897.634789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829897.634793:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829897.634798:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333417216:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829897.634799:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829897.634810:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333417856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634818:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634820:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829897.634822:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634825:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00100000:1.0:1787829897.634829:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417344:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829897.634829:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829897.634834:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634841:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829897.634850:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333417984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829897.634861:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829897.634866:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829897.634868:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829897.634870:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829897.634878:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11708 00000100:00080000:3.0:1787829897.634883:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829897.634889:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb223800 x1874663333417344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829897.634890:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829897.634900:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829897.634908:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333417344/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829897.634916:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333418112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829897.634925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.634932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.634938:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.634946:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829897.634971:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.634983:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb220e00 x1874663333417472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.634998:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333417472/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635018:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635021:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.635026:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829897.635044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.635055:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb222680 x1874663333417600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.635068:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333417600/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635077:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635081:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829897.635091:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829897.635109:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.635118:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb221180 x1874663333417728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.635139:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333417728/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635155:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333417728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829897.635162:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829897.635180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.635188:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb223b80 x1874663333417856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.635196:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333417856/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635203:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635206:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.635249:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829897.635263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.635271:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb220a80 x1874663333417984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.635279:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333417984/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635306:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635310:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333417984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.635317:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333418112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829897.635337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829897.635347:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff9057fb221500 x1874663333418112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.635360:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333418112/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.635369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.635372:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829897.635376:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333418112:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829898.018387:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829899.042392:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829900.066518:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829901.090345:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829902.114403:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829902.754314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754321:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829902.754324:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829902.754331:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754338:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754344:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829902.754374:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333418240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754398:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754401:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829902.754402:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829902.754404:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754406:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754408:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754415:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829902.754422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333418368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754429:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754431:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829902.754432:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829902.754437:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333418240:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829902.754437:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754442:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754445:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754447:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829902.754454:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333418496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754460:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754462:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829902.754463:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829902.754465:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754467:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754470:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754473:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829902.754478:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333418624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754484:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754486:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829902.754487:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829902.754490:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11713 00000800:00000100:1.0:1787829902.754494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829902.754495:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754499:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754502:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829902.754509:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333418752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754514:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754516:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829902.754518:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829902.754520:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754522:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829902.754537:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb221500 x1874663333418240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829902.754538:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754543:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829902.754555:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333418880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829902.754558:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333418240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829902.754574:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754578:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829902.754580:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829902.754580:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829902.754587:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829902.754588:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00100000:1.0:1787829902.754592:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333418240:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829902.754592:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754598:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754601:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829902.754607:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333419008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829902.754612:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829902.754614:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829902.754615:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829902.754617:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418368:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829902.754620:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11713 00000100:00080000:3.0:1787829902.754624:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829902.754627:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829902.754630:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829902.754637:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333419136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829902.754642:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754656:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb220a80 x1874663333418368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754665:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333418368/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754670:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754672:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754674:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.754677:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829902.754687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754692:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb223b80 x1874663333418496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754698:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333418496/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754704:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754706:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.754709:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829902.754718:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754723:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb221180 x1874663333418624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754729:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333418624/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754734:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754735:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754737:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829902.754739:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829902.754748:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754752:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb222680 x1874663333418752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754758:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333418752/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754762:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754764:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754765:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333418752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829902.754768:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333418880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829902.754776:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754781:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb220e00 x1874663333418880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754786:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333418880/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754800:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754802:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754804:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333418880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.754807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829902.754817:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754822:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb223800 x1874663333419008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754828:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333419008/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754834:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754835:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.754847:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829902.754856:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829902.754862:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff9057fb220000 x1874663333419136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.754868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333419136/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.754872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.754873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829902.754875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419136:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829903.138371:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829904.162386:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829905.186337:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829906.210354:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829907.234439:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829907.874303:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874313:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829907.874317:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874325:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874335:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874342:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829907.874377:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333419264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874402:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874406:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829907.874407:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874411:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874415:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874419:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874428:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829907.874434:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333419264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829907.874458:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333419392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874469:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874473:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829907.874475:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874479:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874506:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874512:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829907.874524:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829907.874529:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333419520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874542:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874546:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829907.874548:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829907.874554:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419520:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829907.874554:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874562:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874567:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874573:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829907.874584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333419648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874606:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874609:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829907.874611:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874616:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874621:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829907.874624:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419648:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829907.874628:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829907.874650:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333419776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874661:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874665:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829907.874667:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874672:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00100000:1.0:1787829907.874674:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419776:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829907.874678:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874684:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829907.874703:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333419904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874719:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874723:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829907.874725:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874729:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874733:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829907.874735:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419904:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829907.874740:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874746:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829907.874758:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333420032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829907.874770:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829907.874773:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829907.874775:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829907.874780:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11718 00000100:00080000:3.0:1787829907.874783:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829907.874788:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829907.874792:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829907.874795:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829907.874805:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333420160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829907.874832:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420160:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829908.258399:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829909.282396:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829910.306408:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829910.946514:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1787829910.946556:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829910.946571:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946609:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946617:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946623:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946627:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946631:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946636:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1787829910.946639:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1787829910.946698:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb220380 x1874663333420160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946720:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333420160/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946730:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946735:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946739:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829910.946746:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb222d80 x1874663333420032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946766:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333420032/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946776:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.946785:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb223100 x1874663333419904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946795:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333419904/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946801:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333419904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.946813:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb221880 x1874663333419776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946824:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333419776/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946837:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829910.946844:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb221c00 x1874663333419648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333419648/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946865:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946867:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829910.946872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb222a00 x1874663333419520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946883:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333419520/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946889:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946892:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946895:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.946900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb220700 x1874663333419392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946909:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333419392/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946916:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946919:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946922:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333419392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.946927:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829910] req@ffff9057fb222300 x1874663333419264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.946938:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333419264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.946945:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.946948:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829910.946951:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333419264:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1787829911.330404:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829912.354383:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829912.994291:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994301:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829912.994304:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994313:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994323:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994330:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829912.994364:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333420288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829912.994386:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994389:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829912.994391:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994394:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994398:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994401:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994408:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829912.994422:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333420416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829912.994439:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333420288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829912.994439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994447:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829912.994450:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994470:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994474:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994479:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994484:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829912.994497:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333420544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829912.994508:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994511:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829912.994513:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829912.994519:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829912.994521:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994526:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994530:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994535:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829912.994547:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333420672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829912.994556:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994558:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829912.994561:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994566:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994571:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994574:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994578:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829912.994589:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333420800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829912.994597:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994600:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829912.994601:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994604:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994607:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994610:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994615:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829912.994623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333420928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829912.994630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829912.994632:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb222300 x1874663333420288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829912.994634:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829912.994637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829912.994644:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994649:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994655:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994659:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829912.994662:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333420288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.994678:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829912.994681:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333421056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829912.994684:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.994692:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333420288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829912.994712:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829912.994715:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829912.994717:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829912.994720:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420416:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829912.994724:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11723 00000100:00080000:3.0:1787829912.994730:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829912.994733:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829912.994739:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829912.994749:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829912.994754:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333421184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829912.994766:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb220700 x1874663333420416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.994780:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333420416/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.994788:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.994792:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.994796:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829912.994802:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829912.994819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.994830:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb222a00 x1874663333420544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.994841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333420544/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.994848:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.994851:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.994854:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829912.994860:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829912.994879:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.994889:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb221c00 x1874663333420672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.994899:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333420672/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.994907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.994910:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.994912:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829912.994917:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829912.994934:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.994943:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb221880 x1874663333420800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.994953:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333420800/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.994960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.994962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.994965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333420800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829912.994970:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829912.994986:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.994995:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb223100 x1874663333420928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.995006:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333420928/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.995012:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.995015:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.995019:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333420928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829912.995025:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829912.995043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.995053:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb222d80 x1874663333421056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.995063:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333421056/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.995070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.995073:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.995076:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829912.995095:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829912.995119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829912.995129:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff9057fb220380 x1874663333421184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.995140:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333421184/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.995148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829912.995150:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829912.995153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421184:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829913.378371:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829914.402400:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829915.426341:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829916.450401:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829917.474358:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829918.114392:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114407:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829918.114412:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114441:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114451:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114459:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829918.114500:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333421312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829918.114529:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829918.114534:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114539:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114543:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114547:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114557:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829918.114571:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333421440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829918.114580:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333421312:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829918.114598:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114601:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829918.114603:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114607:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114610:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114615:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114620:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829918.114633:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333421568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829918.114644:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114647:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829918.114650:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114654:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114657:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114661:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114668:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829918.114679:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333421696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829918.114683:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829918.114693:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114697:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829918.114699:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114705:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114709:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114713:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114717:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829918.114727:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333421824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829918.114736:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114740:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829918.114741:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78abb80 x1874663333421312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.114743:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114754:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114758:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114762:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829918.114765:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333421312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829918.114769:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829918.114776:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.114779:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.114781:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333421312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829918.114797:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333421952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829918.114808:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829918.114811:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421440:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829918.114814:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829918.114818:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.114823:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114827:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829918.114833:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829918.114834:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114841:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829918.114847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78a9f80 x1874663333421440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.114856:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333422080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829918.114860:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333421440/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.114872:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.114877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.114882:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829918.114892:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421568:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829918.114900:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.114905:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829918.114908:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829918.114908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829918.114915:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11728 00000100:00080000:1.0:1787829918.114919:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.114923:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829918.114928:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829918.114934:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78a9880 x1874663333421568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.114943:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333422208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829918.114969:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333421568/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.114982:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.114986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.114989:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829918.114994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829918.115016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829918.115025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78a8000 x1874663333421696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.115035:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333421696/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.115041:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.115044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.115047:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829918.115052:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829918.115067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829918.115076:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78ab800 x1874663333421824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.115086:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333421824/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.115099:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.115101:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.115105:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333421824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829918.115110:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829918.115128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829918.115137:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78ab480 x1874663333421952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.115144:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333421952/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.115149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.115151:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.115153:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333421952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829918.115165:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829918.115177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829918.115183:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78aaa00 x1874663333422080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.115189:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333422080/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.115193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.115195:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.115197:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829918.115200:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829918.115209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829918.115214:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9057c78a8380 x1874663333422208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.115255:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333422208/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.115260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.115262:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829918.115264:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422208:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829918.498368:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829919.522390:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829920.546339:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829921.570498:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829922.594425:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829923.234362:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234373:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829923.234376:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234400:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234412:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234423:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829923.234468:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333422336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234491:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234493:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829923.234495:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234498:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234500:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234502:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234511:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829923.234522:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333422464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829923.234533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234536:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234538:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234540:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829923.234547:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333422336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829923.234551:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333422592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234561:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829923.234562:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234565:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234566:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234568:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234572:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829923.234578:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333422720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234583:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234585:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829923.234587:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234589:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234592:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234594:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234597:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829923.234603:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333422848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234608:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234610:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829923.234610:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234614:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234616:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234620:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787829923.234622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234624:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829923.234633:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333422976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234638:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234640:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829923.234641:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829923.234645:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234647:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234649:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234651:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829923.234658:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333423104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829923.234664:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829923.234666:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829923.234669:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829923.234669:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78aad80 x1874663333422336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829923.234676:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11733 00000100:00080000:1.0:1787829923.234680:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829923.234685:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829923.234688:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829923.234693:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333422336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829923.234697:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333423232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829923.234708:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234713:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234717:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333422336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829923.234738:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829923.234754:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.234762:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78a8380 x1874663333422464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.234770:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333422464/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.234775:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234777:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234779:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829923.234782:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829923.234792:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.234797:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78aaa00 x1874663333422592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.234804:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333422592/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.234809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234810:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829923.234815:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829923.234837:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.234848:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78ab480 x1874663333422720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.234866:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333422720/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.234874:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234877:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234880:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829923.234886:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829923.234905:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.234916:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78ab800 x1874663333422848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.234924:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333422848/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.234930:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234934:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333422848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829923.234937:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829923.234949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.234955:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78a8000 x1874663333422976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.234961:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333422976/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.234977:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.234979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.234983:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333422976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829923.234987:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333423104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829923.234998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.235004:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78a9880 x1874663333423104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.235010:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333423104/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.235015:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.235016:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.235018:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333423104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829923.235021:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333423232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829923.235032:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829923.235037:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829923/real 1787829923] req@ffff9057c78a9f80 x1874663333423232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829923.235043:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333423232/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829923.235048:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829923.235049:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829923.235052:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333423232:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829923.618421:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829924.642476:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829925.666351:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829926.690387:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829927.714400:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829928.354315:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354322:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829928.354325:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354332:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354340:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829928.354346:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829928.354386:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333423360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829928.354409:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354412:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829928.354413:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354415:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354418:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829928.354421:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354428:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829928.354444:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333423488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829928.354451:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333423360:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829928.354454:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354459:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829928.354461:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354463:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829928.354467:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354470:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829928.354478:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333423616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829928.354484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354488:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829928.354490:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354493:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354497:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829928.354500:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354505:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829928.354516:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829928.354519:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333423744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829928.354535:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354538:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829928.354540:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354544:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354548:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829928.354551:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78abb80 x1874663333423360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829928.354556:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354565:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829928.354572:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333423360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829928.354580:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333423872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829928.354581:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.354592:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829928.354595:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829928.354596:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333423360:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829928.354604:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829928.354608:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354613:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:1.0:1787829928.354617:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829928.354621:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423488:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829928.354624:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354632:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829928.354642:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829928.354648:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333424000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829928.354653:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78aa680 x1874663333423488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829928.354673:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829928.354675:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333423488/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829928.354678:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829928.354683:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829928.354688:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829928.354692:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:2.0:1787829928.354695:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.354700:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423488:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829928.354701:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829928.354706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423616:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829928.354710:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354719:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829928.354727:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829928.354735:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333424128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829928.354737:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78a9c00 x1874663333423616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829928.354748:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829928.354753:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829928.354755:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333423616/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829928.354757:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829928.354766:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11739 00000100:00080000:2.0:1787829928.354768:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829928.354774:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829928.354774:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.354781:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423616:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829928.354783:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829928.354792:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829928.354794:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829928.354807:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333424256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829928.354819:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829928.354832:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78aa300 x1874663333423744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829928.354841:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333423744/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829928.354846:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.354849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.354851:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829928.354855:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829928.354866:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829928.354872:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78a8a80 x1874663333423872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829928.354890:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333423872/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829928.354897:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.354900:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.354903:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333423872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829928.354918:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829928.354933:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829928.354940:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78a9180 x1874663333424000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829928.354946:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333424000/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829928.354951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.354953:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.354954:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829928.354959:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829928.354974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829928.354984:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78a8700 x1874663333424128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829928.354995:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333424128/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829928.355007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.355009:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.355011:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829928.355014:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829928.355031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829928.355041:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829928/real 1787829928] req@ffff9057c78a9500 x1874663333424256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829928.355051:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333424256/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829928.355059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829928.355062:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829928.355065:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333424256:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829928.738465:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829929.762335:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829930.786430:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829931.810384:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829932.834396:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829933.474347:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474354:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829933.474357:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474364:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474372:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829933.474377:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829933.474409:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333424384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829933.474436:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474444:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829933.474445:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474450:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474453:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829933.474455:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829933.474462:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829933.474472:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333424512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829933.474480:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474482:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829933.474484:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474487:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474490:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829933.474496:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333424384:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829933.474496:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829933.474501:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829933.474517:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333424640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829933.474526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474530:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829933.474531:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474535:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474549:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829933.474552:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829933.474557:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829933.474558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829933.474571:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333424768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829933.474579:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474582:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829933.474584:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474588:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474591:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829933.474596:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474597:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb220000 x1874663333424384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829933.474602:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829933.474616:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333424896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829933.474622:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333424384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829933.474630:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474637:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829933.474639:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829933.474641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829933.474643:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474648:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829933.474649:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829933.474654:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474659:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333424384:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1787829933.474661:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829933.474674:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333425024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829933.474683:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424512:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829933.474685:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829933.474689:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829933.474691:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829933.474695:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474703:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829933.474706:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829933.474710:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829933.474716:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829933.474720:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb223800 x1874663333424512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829933.474731:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333425152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829933.474732:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333424512/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474760:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.474763:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829933.474763:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829933.474769:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424512:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829933.474771:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829933.474775:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829933.474780:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424640:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829933.474781:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11744 00000100:00080000:3.0:1787829933.474788:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829933.474793:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829933.474798:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829933.474807:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829933.474812:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333425280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829933.474822:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb220e00 x1874663333424640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.474834:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333424640/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474839:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.474841:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.474848:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829933.474853:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829933.474867:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474873:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb222680 x1874663333424768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.474880:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333424768/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474884:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.474886:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.474888:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829933.474891:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829933.474900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474905:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb221180 x1874663333424896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.474911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333424896/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.474917:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.474918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333424896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829933.474930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829933.474947:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb223b80 x1874663333425024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.474964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333425024/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.474969:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.474971:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829933.474974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829933.474984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829933.474989:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb220a80 x1874663333425152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.474995:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333425152/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.474999:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.475000:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.475002:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829933.475005:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829933.475013:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829933.475018:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829933/real 1787829933] req@ffff9057fb221500 x1874663333425280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829933.475023:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333425280/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829933.475031:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829933.475032:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829933.475034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333425280:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829933.858391:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829934.882439:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829935.906473:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829936.930404:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829937.954402:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829938.594357:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594365:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829938.594368:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594376:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594385:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594390:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829938.594422:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333425408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594446:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594448:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829938.594449:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594451:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594453:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594456:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594462:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829938.594470:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333425536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594476:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594479:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829938.594480:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594483:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594485:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594487:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594490:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829938.594494:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333425408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829938.594499:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333425664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594506:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594508:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829938.594509:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594511:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594513:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594515:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594518:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829938.594525:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333425792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594530:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594532:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829938.594533:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594536:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594539:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594541:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594545:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829938.594550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829938.594554:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333425920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594572:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829938.594578:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594582:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594586:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829938.594592:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de0a80 x1874663333425408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829938.594592:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594602:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829938.594611:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333425408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829938.594617:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333426048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829938.594621:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594627:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333425408:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829938.594636:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829938.594641:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425536:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829938.594642:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829938.594647:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594652:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594657:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594661:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787829938.594664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787829938.594665:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829938.594676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de1180 x1874663333425536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829938.594677:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333426176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829938.594704:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829938.594705:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829938.594707:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829938.594710:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11749 00000100:00080000:2.0:1787829938.594716:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829938.594721:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594724:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333425536/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829938.594726:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829938.594737:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594740:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829938.594744:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333426304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829938.594745:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.594776:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829938.594793:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594800:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de1880 x1874663333425664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333425664/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.594812:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594814:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594816:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.594820:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829938.594836:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594847:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de3800 x1874663333425792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594858:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333425792/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.594866:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594868:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594870:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829938.594874:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829938.594888:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594894:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de3100 x1874663333425920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594900:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333425920/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.594905:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594908:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333425920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829938.594911:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829938.594920:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594931:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de2a00 x1874663333426048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594938:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333426048/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.594943:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594945:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594946:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.594950:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829938.594960:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594965:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de0700 x1874663333426176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594971:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333426176/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.594975:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.594977:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.594978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.594981:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829938.594989:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829938.594994:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff9057d0de1c00 x1874663333426304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.594999:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333426304/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.595004:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.595005:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829938.595008:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333426304:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829938.978404:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829940.002385:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829941.026452:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829942.050356:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829943.074514:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829943.714308:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714315:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829943.714318:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714334:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714344:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714352:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829943.714388:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333426432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829943.714411:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714413:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829943.714415:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714418:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714421:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714425:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714432:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829943.714455:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333426560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829943.714455:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333426432:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829943.714462:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714465:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829943.714466:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714469:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714474:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714477:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714480:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829943.714489:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333426688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829943.714495:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714498:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829943.714499:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714502:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714504:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714505:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714509:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829943.714515:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333426816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829943.714520:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714523:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829943.714524:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714526:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714528:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714529:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714532:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829943.714536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829943.714542:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333426944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829943.714548:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714551:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829943.714553:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714556:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714558:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714560:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714563:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829943.714569:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333427072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829943.714574:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714576:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829943.714577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829943.714581:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78ab100 x1874663333426432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.714582:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714586:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714590:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829943.714595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829943.714605:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333427200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829943.714612:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333426432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.714614:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829943.714619:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829943.714622:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829943.714628:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11754 00000100:00080000:1.0:1787829943.714633:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829943.714636:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1787829943.714637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829943.714641:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829943.714645:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829943.714650:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333427328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829943.714653:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333426432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829943.714681:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829943.714702:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.714713:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78a9500 x1874663333426560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.714725:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333426560/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.714733:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.714736:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.714739:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829943.714745:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829943.714760:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.714768:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78a8700 x1874663333426688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.714778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333426688/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.714785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.714788:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.714792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829943.714807:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829943.714825:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.714835:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78a9180 x1874663333426816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.714845:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333426816/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.714852:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.714856:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.714859:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829943.714865:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829943.714882:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.714893:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78a8a80 x1874663333426944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.714904:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333426944/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.714912:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.714915:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.714918:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333426944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829943.714923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829943.714939:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.714948:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78aa300 x1874663333427072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.714960:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333427072/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.714967:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.714970:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.714974:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829943.714979:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829943.714993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.715002:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78a9c00 x1874663333427200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.715012:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333427200/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.715019:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.715022:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.715025:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829943.715046:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829943.715063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829943.715073:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff9057c78aa680 x1874663333427328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829943.715083:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333427328/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829943.715090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829943.715093:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829943.715096:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333427328:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829944.098650:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829945.122361:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829946.146349:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829947.170402:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829948.194341:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829948.834322:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834329:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829948.834332:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834340:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834348:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834354:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829948.834386:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333427456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834406:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834408:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829948.834409:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834412:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834414:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834417:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834423:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829948.834430:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333427584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834437:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834439:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829948.834440:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834443:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834449:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829948.834453:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333427456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1787829948.834459:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333427712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834466:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829948.834469:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834471:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834473:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834475:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829948.834484:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333427840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834490:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834493:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829948.834494:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834496:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834505:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834507:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834510:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829948.834515:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829948.834519:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333427968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834526:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834528:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829948.834530:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834532:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834535:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834538:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834540:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829948.834546:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333428096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829948.834552:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834555:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829948.834558:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb221500 x1874663333427456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829948.834558:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834561:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834566:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834571:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834576:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829948.834584:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333428224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829948.834590:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333427456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829948.834592:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829948.834596:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829948.834598:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829948.834604:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11759 00000100:00080000:3.0:1787829948.834608:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829948.834612:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829948.834616:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829948.834619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829948.834623:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333428352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829948.834628:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834637:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333427456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829948.834648:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829948.834682:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834694:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb220a80 x1874663333427584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.834707:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333427584/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.834715:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.834718:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834721:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829948.834743:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829948.834759:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834769:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb223b80 x1874663333427712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.834778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333427712/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.834787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.834789:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834792:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829948.834798:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829948.834815:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834826:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb221180 x1874663333427840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.834838:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333427840/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.834846:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.834849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834852:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829948.834858:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829948.834878:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834887:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb222680 x1874663333427968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.834897:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333427968/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.834903:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.834906:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834909:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333427968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829948.834913:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829948.834930:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834939:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb220e00 x1874663333428096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.834952:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333428096/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.834959:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.834962:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.834965:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829948.834971:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829948.834988:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.834996:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb223800 x1874663333428224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.835002:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333428224/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.835007:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.835009:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.835010:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829948.835013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829948.835023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829948.835028:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829948] req@ffff9057fb220000 x1874663333428352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829948.835034:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333428352/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829948.835042:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829948.835044:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829948.835046:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333428352:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829949.218375:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829950.242487:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829951.266357:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829952.290397:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829953.314424:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829953.954298:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954306:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829953.954308:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954315:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954323:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829953.954328:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829953.954356:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333428480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829953.954376:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954380:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829953.954382:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954386:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954390:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829953.954394:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954403:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829953.954412:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333428480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829953.954418:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333428608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829953.954430:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954434:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829953.954436:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954440:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954443:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829953.954446:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954450:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829953.954458:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333428736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829953.954464:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954467:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829953.954469:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829953.954470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829953.954474:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954477:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829953.954479:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954482:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829953.954489:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333428864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829953.954494:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954496:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829953.954498:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954501:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954504:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829953.954506:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954509:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829953.954510:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de1500 x1874663333428480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829953.954519:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333428992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829953.954524:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954526:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829953.954527:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954530:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954532:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829953.954535:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333428480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829953.954536:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954542:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829953.954550:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829953.954552:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333429120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829953.954555:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954558:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333428480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829953.954575:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428608:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829953.954576:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954581:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829953.954583:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829953.954588:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954593:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829953.954597:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829953.954601:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954607:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829953.954609:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de0000 x1874663333428608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954624:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333428608/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954632:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954633:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829953.954634:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333429248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829953.954640:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829953.954652:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829953.954655:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829953.954658:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829953.954659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829953.954666:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11764 00000100:00080000:2.0:1787829953.954670:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829953.954676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de0e00 x1874663333428736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829953.954676:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829953.954685:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829953.954690:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333428736/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954696:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829953.954697:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333429376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829953.954700:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954704:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829953.954718:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829953.954741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829953.954752:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de1f80 x1874663333428864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954760:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333428864/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954764:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954766:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954768:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829953.954771:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829953.954783:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829953.954789:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de1c00 x1874663333428992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954796:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333428992/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954804:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954807:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333428992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829953.954830:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829953.954852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829953.954860:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de0700 x1874663333429120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954867:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333429120/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954873:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954875:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829953.954878:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829953.954891:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829953.954900:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de2a00 x1874663333429248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954911:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333429248/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954930:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829953.954934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829953.954949:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829953.954958:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff9057d0de3100 x1874663333429376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.954968:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333429376/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.954976:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.954979:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829953.954982:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333429376:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829954.338412:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829955.362382:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829956.386366:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829957.410452:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829958.434395:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829959.074333:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074342:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829959.074346:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074356:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074366:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074374:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829959.074420:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333429504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074451:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074455:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829959.074457:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074461:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074465:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074469:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074478:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.074490:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333429632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074500:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074504:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.074505:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074509:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074512:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074515:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074519:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.074528:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333429760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074536:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074539:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829959.074541:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074551:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00100000:1.0:1787829959.074552:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333429504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829959.074555:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074560:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074565:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829959.074577:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333429888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074587:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074590:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.074592:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074596:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074601:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074605:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074610:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829959.074631:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829959.074633:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333430016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074645:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074649:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829959.074651:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074655:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074660:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074665:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074670:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829959.074676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de3800 x1874663333429504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829959.074686:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333430144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074697:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829959.074699:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333429504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829959.074703:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.074707:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.074717:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:1.0:1787829959.074721:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829959.074722:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074730:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787829959.074731:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:2.0:1787829959.074739:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829959.074740:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333429504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829959.074752:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829959.074756:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333430272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.074771:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.074774:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.074776:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829959.074781:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829959.074782:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11769 00000100:00080000:2.0:1787829959.074788:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.074792:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829959.074797:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829959.074798:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de1880 x1874663333429632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829959.074810:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333430400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829959.074817:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333429632/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.074829:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.074833:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.074836:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.074843:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829959.074864:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.074875:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de1180 x1874663333429760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.074886:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333429760/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.074894:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.074896:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.074900:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.074922:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.074941:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.074953:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de0a80 x1874663333429888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.074964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333429888/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.074971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.074973:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.074977:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333429888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829959.074983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.075000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.075009:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de3100 x1874663333430016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.075019:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333430016/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.075026:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.075029:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.075032:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829959.075036:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829959.075053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.075062:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de2a00 x1874663333430144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.075072:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333430144/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.075079:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.075082:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.075084:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.075090:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829959.075113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.075123:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de0700 x1874663333430272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.075147:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333430272/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.075155:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.075158:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.075161:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.075167:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.075186:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829959.075196:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9057d0de1c00 x1874663333430400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.075206:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333430400/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.075211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.075212:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829959.075215:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333430400:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829959.458429:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829960.482770:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829961.506418:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829962.530385:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829963.554456:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829964.196204:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196214:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829964.196253:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829964.196262:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196273:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196280:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829964.196316:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333430528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829964.196354:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196357:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829964.196359:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829964.196363:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196367:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196371:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829964.196382:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829964.196395:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333430528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829964.196396:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333430656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829964.196411:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196414:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829964.196416:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829964.196419:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196422:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196425:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829964.196429:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829964.196440:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333430784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829964.196448:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196451:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829964.196453:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829964.196456:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196460:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829964.196464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829964.196467:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829964.196473:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829964.196485:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333430912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829964.196498:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196501:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829964.196502:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829964.196506:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de1f80 x1874663333430528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829964.196510:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196516:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196522:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829964.196526:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333430528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829964.196527:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829964.196541:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829964.196543:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333431040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829964.196548:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196555:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333430528:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829964.196559:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196563:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829964.196565:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829964.196570:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196573:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196577:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829964.196582:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430656:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1787829964.196582:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829964.196606:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333431168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829964.196612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829964.196621:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196625:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829964.196627:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829964.196631:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de0e00 x1874663333430656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829964.196632:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196638:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829964.196652:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829964.196656:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333430656/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829964.196666:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333431296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829964.196667:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.196676:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196684:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430656:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829964.196685:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829964.196691:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829964.196693:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829964.196695:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430784:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787829964.196700:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11774 00000100:00080000:2.0:1787829964.196704:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829964.196708:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829964.196712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829964.196722:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829964.196725:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333431424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829964.196739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de0000 x1874663333430784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.196753:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333430784/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.196761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.196765:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196768:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829964.196773:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829964.196792:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829964.196803:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de1500 x1874663333430912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.196814:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333430912/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.196822:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.196825:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196828:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333430912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829964.196834:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829964.196858:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829964.196882:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de1c00 x1874663333431040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.196893:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333431040/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.196901:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.196904:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196907:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829964.196923:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829964.196944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829964.196954:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de0700 x1874663333431168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.196964:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333431168/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.196972:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.196975:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.196978:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829964.196983:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829964.197000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829964.197011:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de2a00 x1874663333431296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.197021:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333431296/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.197028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.197031:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.197034:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829964.197040:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829964.197058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829964.197068:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829964] req@ffff9057d0de3100 x1874663333431424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829964.197079:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333431424/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829964.197086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829964.197089:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829964.197092:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333431424:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829964.578489:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829965.602404:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829966.626350:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829967.650409:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829968.674380:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829969.314524:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314533:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829969.314537:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314545:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314555:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314561:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829969.314596:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333431552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314623:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314626:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829969.314628:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314632:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314636:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314640:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314648:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829969.314660:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333431552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829969.314663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333431680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314679:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829969.314680:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314684:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314687:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314691:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314695:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829969.314705:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333431808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314714:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314717:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829969.314718:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314721:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314725:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314728:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314733:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829969.314742:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333431936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314755:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314758:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829969.314759:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829969.314765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829969.314766:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314772:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314775:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314780:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829969.314790:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333432064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314800:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829969.314803:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829969.314805:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314809:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314812:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314816:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314821:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829969.314822:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78abb80 x1874663333431552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.314835:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333432192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829969.314844:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829969.314846:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333431552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.314849:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829969.314852:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314856:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314860:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314866:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314875:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829969.314876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829969.314886:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333432320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829969.314893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829969.314900:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829969.314902:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333431552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829969.314905:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829969.314910:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829969.314915:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11780 00000100:00080000:1.0:1787829969.314919:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829969.314922:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829969.314926:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829969.314933:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829969.314939:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333432448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829969.314963:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.314979:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78a9f80 x1874663333431680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.314991:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333431680/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315003:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315006:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829969.315013:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829969.315031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315042:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78a9880 x1874663333431808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315053:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333431808/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315074:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315077:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829969.315083:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829969.315102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315122:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78a8000 x1874663333431936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315133:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333431936/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315143:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315147:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333431936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829969.315152:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829969.315167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315175:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78ab800 x1874663333432064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315185:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333432064/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315196:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315207:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829969.315213:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829969.315251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315262:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78ab480 x1874663333432192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315272:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333432192/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315281:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315283:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829969.315288:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829969.315314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315325:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78aaa00 x1874663333432320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315335:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333432320/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315345:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315347:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829969.315365:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829969.315383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829969.315392:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829969/real 1787829969] req@ffff9057c78a8380 x1874663333432448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829969.315402:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333432448/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829969.315408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829969.315411:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829969.315413:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333432448:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829969.698693:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829970.722550:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829971.746496:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829972.770492:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829973.794389:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829974.434338:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434344:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829974.434347:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434355:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434363:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434369:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829974.434403:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333432576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829974.434426:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434430:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829974.434432:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434436:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434439:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434442:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434449:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829974.434461:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333432704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829974.434471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333432576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829974.434472:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434480:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829974.434483:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434487:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434490:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434494:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434499:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829974.434512:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333432832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829974.434518:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434521:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829974.434522:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434524:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434527:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434530:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434535:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829974.434545:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333432960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829974.434568:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829974.434571:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432704:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829974.434573:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829974.434577:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434583:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434587:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434590:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434595:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829974.434600:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829974.434613:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333433088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829974.434628:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432960:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787829974.434629:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434635:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829974.434636:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434640:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434643:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434646:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829974.434650:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433088:192.168.203.156@tcp:8:kworker.0 00000008:00100000:3.0:1787829974.434652:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829974.434661:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333433216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829974.434667:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434669:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829974.434671:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434673:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434676:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434679:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434682:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829974.434686:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829974.434692:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333433344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829974.434701:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829974.434703:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829974.434704:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829974.434706:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11785 00000100:00080000:3.0:1787829974.434715:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829974.434717:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829974.434720:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829974.434726:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829974.434732:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333433472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829974.434751:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433472:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829974.820653:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829975.842409:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829976.866323:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829977.507301:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1787829977.507343:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829977.507358:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508195:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508208:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508216:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829977.508319:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb222d80 x1874663333432704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.508343:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333432704/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.508354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.508359:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.508363:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829977.508371:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb223100 x1874663333432832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.508383:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333432832/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.508391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.508394:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.508398:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829977.508404:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb220380 x1874663333432576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.508416:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333432576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.508424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.508427:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.508430:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333432576:192.168.203.156@tcp:250:kworker.0 00000800:00000100:0.0:1787829977.508901:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508911:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508918:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1787829977.508925:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1787829977.508981:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb222300 x1874663333433472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.509615:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333433472/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.509627:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.509632:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.509636:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829977.509643:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb220700 x1874663333433344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.509657:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333433344/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.509664:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.509667:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.509670:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829977.509676:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb222a00 x1874663333433216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.509687:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333433216/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.509695:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.509699:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.509702:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333433216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829977.509708:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb221c00 x1874663333433088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.509719:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333433088/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.509726:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.509730:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.509732:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829977.509737:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829974/real 1787829977] req@ffff9057fb221880 x1874663333432960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829977.509749:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333432960/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829977.509757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829977.509760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829977.509764:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333432960:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829977.890420:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829978.914592:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829979.554329:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554338:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829979.554342:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554353:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554362:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829979.554370:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787829979.554403:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333433600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554434:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554448:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829979.554449:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554454:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829979.554462:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554471:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829979.554483:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333433728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554493:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554497:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829979.554499:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554502:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554506:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829979.554509:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333433600:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829979.554512:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554526:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829979.554539:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333433856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554549:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554552:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829979.554554:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554557:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554562:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829979.554565:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554571:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829979.554572:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787829979.554586:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333433984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554596:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554599:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829979.554601:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554605:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554608:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829979.554612:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb221880 x1874663333433600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829979.554615:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554623:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829979.554638:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333433600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829979.554638:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333434112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554670:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829979.554671:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829979.554678:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829979.554680:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829979.554681:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829979.554689:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333433600:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787829979.554689:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:3.0:1787829979.554697:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829979.554703:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433728:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787829979.554704:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554712:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829979.554726:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333434240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829979.554736:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787829979.554737:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554742:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829979.554745:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554750:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00100000:2.0:1787829979.554755:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb221c00 x1874663333433728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829979.554755:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829979.554762:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554771:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829979.554778:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333433728/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829979.554785:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333434368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829979.554815:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829979.554819:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829979.554821:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829979.554824:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11790 00000100:00080000:2.0:1787829979.554825:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829979.554827:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829979.554835:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787829979.554835:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787829979.554844:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829979.554845:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1787829979.554858:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333434496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829979.554875:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829979.554900:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.554911:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb222a00 x1874663333433856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.554922:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333433856/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.554929:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.554932:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.554935:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829979.554941:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829979.554959:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.554968:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb220700 x1874663333433984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.554979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333433984/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.554986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.554989:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.555261:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333433984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829979.555275:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829979.555297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.555307:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb222300 x1874663333434112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.555317:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333434112/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.555323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.555326:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.555328:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829979.555333:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829979.555353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.555363:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb220380 x1874663333434240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.555373:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333434240/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.555380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.555383:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.555386:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829979.555390:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829979.555404:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.555413:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb223100 x1874663333434368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.555422:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333434368/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.555428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.555430:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.555433:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787829979.555437:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829979.555451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829979.555458:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829979/real 1787829979] req@ffff9057fb222d80 x1874663333434496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829979.555468:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333434496/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829979.555474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829979.555477:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829979.555479:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333434496:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829979.938765:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829980.962625:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829981.986414:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829983.010537:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829984.034393:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829984.674744:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.674754:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829984.674758:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.674766:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.674780:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.674787:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829984.674857:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aad80 x1874663333434624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829984.674886:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.674889:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829984.674891:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.674895:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.674899:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.674903:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.674911:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829984.674924:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8380 x1874663333434752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829984.674930:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333434624:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787829984.674934:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.674936:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787829984.675344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.675382:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78aad80 x1874663333434624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.675407:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aad80 x1874663333434624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.675416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.675420:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.675423:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333434624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787829984.675458:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829984.675480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.675492:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78a8380 x1874663333434752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.675504:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8380 x1874663333434752/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.675512:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.675515:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.675518:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434752:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829984.675753:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.675759:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.675763:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.675767:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.675773:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829984.675791:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aaa00 x1874663333434880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829984.675809:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.675813:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829984.675814:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.675817:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.675820:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.675823:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.675828:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829984.675839:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab480 x1874663333435008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829984.676146:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829984.676193:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.676205:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78aaa00 x1874663333434880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.676377:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aaa00 x1874663333434880/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.676391:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829984.676393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.676394:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829984.676395:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829984.676397:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:1.0:1787829984.676399:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00100000:2.0:1787829984.676400:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333434880:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829984.676403:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.676407:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.676412:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829984.676412:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829984.676437:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab800 x1874663333435136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829984.676446:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.676449:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829984.676451:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.676455:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.676458:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.676462:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.676467:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829984.676476:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8000 x1874663333435264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829984.676484:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.676487:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829984.676489:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829984.676490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1787829984.676492:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.676495:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829984.676498:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829984.676502:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78ab480 x1874663333435008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829984.676503:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829984.676513:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab480 x1874663333435008/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.676521:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.676524:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.676527:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829984.676539:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829984.676558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.676569:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78ab800 x1874663333435136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.676580:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab800 x1874663333435136/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.676588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.676591:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.676595:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787829984.676600:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829984.676619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.676629:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78a8000 x1874663333435264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.676640:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8000 x1874663333435264/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.676647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.676650:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.676653:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829984.677180:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9880 x1874663333435392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829984.677201:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829984.677204:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829984.677206:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829984.677209:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11795 00000100:00080000:1.0:1787829984.677213:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829984.677352:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787829984.677377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.677390:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78a9880 x1874663333435392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.677402:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9880 x1874663333435392/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.677410:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.677413:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.677416:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435392:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1787829984.677419:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787829984.677426:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829984.677440:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9f80 x1874663333435520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829984.677806:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787829984.677831:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787829984.677844:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff9057c78a9f80 x1874663333435520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829984.677855:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9f80 x1874663333435520/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829984.677863:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829984.677866:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787829984.677869:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333435520:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829985.059167:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829986.082732:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829987.106539:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829988.130359:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829989.154375:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829989.794373:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.794382:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829989.794386:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.794395:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.794406:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.794413:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829989.794447:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333435648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829989.794505:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333435648:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787829989.794754:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829989.794789:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.794793:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829989.794795:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.794800:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.794804:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.794809:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.794817:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829989.794831:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333435776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829989.795048:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de0a80 x1874663333435648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.795065:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333435648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829989.795086:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795090:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.795092:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795096:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795100:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795104:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795109:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829989.795122:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333435904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.795131:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795134:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829989.795136:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795138:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795141:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795145:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795149:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829989.795159:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333436032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.795168:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795171:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.795173:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795177:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795180:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795183:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795187:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829989.795199:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333436160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.795207:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795210:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829989.795212:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795280:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795284:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795287:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795292:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829989.795302:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333436288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.795310:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795313:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.795315:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795318:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795321:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795324:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795328:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829989.795336:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333436416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.795345:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.795348:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.795349:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.795353:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11800 00000100:00080000:2.0:1787829989.795356:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.795359:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829989.795362:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829989.795371:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333436544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829989.795442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.795446:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.795449:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333435648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829989.795471:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.795499:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.795511:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de1180 x1874663333435776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.795522:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333435776/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.795631:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.795635:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.795638:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.795645:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.795667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.795677:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de1880 x1874663333435904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.795688:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333435904/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.795695:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.795698:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.795701:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333435904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.795707:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829989.795742:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.795753:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de3800 x1874663333436032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.795882:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333436032/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.795890:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.795893:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.795896:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829989.795903:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829989.795935:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.795946:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de3100 x1874663333436160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.795956:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333436160/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.795963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.795966:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.795969:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829989.795974:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.795991:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.796000:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de2a00 x1874663333436288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.796011:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333436288/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.796018:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.796021:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.796024:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.796029:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.796043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.796052:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de0700 x1874663333436416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.796062:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333436416/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.796128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.796131:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.796134:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.796139:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829989.796160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829989.796171:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff9057d0de1c00 x1874663333436544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.796182:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333436544/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.796195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.796198:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829989.796201:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333436544:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787829990.178703:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829991.202383:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829992.226361:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829993.250464:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829994.274382:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829994.914314:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914323:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829994.914327:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914336:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914346:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914351:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787829994.914381:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333436672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829994.914404:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914407:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829994.914408:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914410:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914413:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914415:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914422:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829994.914431:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333436800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829994.914438:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914440:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829994.914441:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914443:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914444:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914446:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914449:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829994.914455:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333436928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829994.914460:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333436672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787829994.914463:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914468:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829994.914470:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914472:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914474:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914476:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914479:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829994.914485:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333437056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829994.914495:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914497:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829994.914498:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914500:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914503:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914505:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914508:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829994.914517:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333437184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829994.914527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787829994.914527:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914533:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829994.914534:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914539:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914542:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914546:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914550:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829994.914559:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333437312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829994.914565:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914567:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829994.914568:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914570:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914575:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914578:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787829994.914582:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829994.914588:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333437440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829994.914594:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829994.914596:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829994.914598:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829994.914600:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11805 00000100:00080000:2.0:1787829994.914603:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829994.914606:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914611:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de1500 x1874663333436672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829994.914612:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829994.914627:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333437568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829994.914637:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333436672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.914649:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.914654:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.914659:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333436672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1787829994.914667:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.914690:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914703:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de0000 x1874663333436800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.914716:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333436800/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.914724:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.914727:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.914730:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.914751:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.914771:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de0e00 x1874663333436928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.914807:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333436928/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.914815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.914819:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.914822:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333436928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.914828:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829994.914847:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914857:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de1f80 x1874663333437056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.914868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333437056/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.914875:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.914878:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.914881:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829994.914899:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829994.914919:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914930:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de1c00 x1874663333437184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.914941:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333437184/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.914949:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.914952:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.914955:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787829994.914961:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.914979:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.914990:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de0700 x1874663333437312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.915001:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333437312/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.915009:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.915012:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.915015:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.915020:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.915039:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.915050:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de2a00 x1874663333437440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.915060:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333437440/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.915074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.915077:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.915080:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.915086:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787829994.915104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787829994.915115:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff9057d0de3100 x1874663333437568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.915126:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333437568/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.915134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.915136:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787829994.915139:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333437568:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787829995.298406:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829996.322402:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829997.346633:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829998.370407:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829999.394411:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830000.034340:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034349:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830000.034353:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830000.034362:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00080000:2.0:1787830000.034371:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034378:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787830000.034412:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333437696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830000.034439:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034443:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830000.034444:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830000.034448:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00080000:2.0:1787830000.034451:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034455:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034463:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830000.034487:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1880 x1874663333437824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830000.034493:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333437696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787830000.034504:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034508:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830000.034509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830000.034514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00080000:2.0:1787830000.034517:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830000.034536:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1180 x1874663333437952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830000.034545:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034548:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830000.034550:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830000.034553:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11810 00000800:00000100:1.0:1787830000.034583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830000.034622:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de3800 x1874663333437696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.034640:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333437696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.034650:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.034654:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.034657:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333437696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1787830000.034666:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034674:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034680:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830000.034689:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830000.034695:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0a80 x1874663333438080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830000.034707:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034710:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830000.034712:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830000.034718:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787830000.034718:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00080000:2.0:1787830000.034724:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034730:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034734:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830000.034736:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de1880 x1874663333437824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830000.034746:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3800 x1874663333438208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830000.034754:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1880 x1874663333437824/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830000.034760:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034768:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830000.034772:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830000.034773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.034778:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1787830000.034785:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00100000:1.0:1787830000.034786:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437824:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787830000.034792:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830000.034797:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437952:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1787830000.034799:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034807:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830000.034820:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333438336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830000.034826:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1787830000.034836:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034839:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787830000.034841:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de1180 x1874663333437952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830000.034842:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830000.034867:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00100000:1.0:1787830000.034872:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1180 x1874663333437952/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830000.034872:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034882:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1787830000.034886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787830000.034889:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830000.034892:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.034897:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333437952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830000.034905:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333438464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830000.034916:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830000.034919:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830000.034921:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830000.034925:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438080:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1787830000.034926:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11810 00000100:00080000:2.0:1787830000.034933:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830000.034938:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830000.034942:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787830000.034954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830000.034963:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333438592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830000.034969:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de0a80 x1874663333438080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.034985:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0a80 x1874663333438080/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.034994:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.035008:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.035012:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830000.035018:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.035037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830000.035048:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de3800 x1874663333438208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.035059:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3800 x1874663333438208/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.035067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.035070:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.035073:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830000.035078:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830000.035095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830000.035105:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de3100 x1874663333438336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.035115:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333438336/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.035123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.035126:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.035129:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.035142:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830000.035159:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830000.035169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de2a00 x1874663333438464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.035180:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333438464/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.035187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.035190:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.035193:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.035198:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.035265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830000.035276:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9057d0de0700 x1874663333438592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.035286:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333438592/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.035293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.035296:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830000.035299:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333438592:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787830000.418552:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830001.442417:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830002.466409:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830003.490644:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830004.514612:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830005.154378:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154422:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830005.154426:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.154435:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.154445:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.154457:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787830005.154491:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333438720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.154515:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154519:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.154520:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.154525:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.154529:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.154533:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.154541:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.154553:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333438848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830005.154555:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333438720:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1787830005.154614:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830005.154651:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb220000 x1874663333438720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.154668:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333438720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.154678:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.154682:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.154686:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333438720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787830005.154691:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154694:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.154696:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.154699:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.154703:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830005.154705:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438848:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787830005.154706:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.154711:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.154723:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333438976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830005.154727:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787830005.154733:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154736:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.154738:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830005.154739:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb223800 x1874663333438848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830005.154742:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.154745:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.154749:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787830005.154750:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333438848/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830005.154753:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787830005.154757:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.154760:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1787830005.154762:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333439104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830005.154763:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.154774:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438976:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787830005.154784:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154787:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.154788:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830005.154791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787830005.154792:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.154795:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.154798:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.154802:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830005.154826:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb220e00 x1874663333438976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.154839:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333438976/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.154846:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.154849:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.154852:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333438976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.154864:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.154894:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830005.154906:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb222680 x1874663333439104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.154918:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333439104/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.154925:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.154928:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.154931:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1787830005.154967:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333439232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.154986:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.154990:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.154992:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.154997:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.155001:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.155005:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.155010:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.155022:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333439360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.155036:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.155039:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.155041:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.155044:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.155048:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.155051:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.155055:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.155065:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333439488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.155074:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830005.155075:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439232:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1787830005.155077:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.155079:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.155082:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11815 00000100:00080000:3.0:1787830005.155084:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.155088:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830005.155092:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787830005.155095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1787830005.155102:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333439616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830005.155106:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb221180 x1874663333439232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.155116:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333439232/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.155124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.155127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.155130:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830005.155135:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.155153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830005.155160:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb223b80 x1874663333439360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.155167:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333439360/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.155172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.155173:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.155175:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.155184:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.155194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830005.155200:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb220a80 x1874663333439488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.155206:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333439488/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.155210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.155211:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.155214:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.155216:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.155244:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830005.155250:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9057fb221500 x1874663333439616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.155256:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333439616/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.155260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.155262:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830005.155264:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333439616:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787830005.538621:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830006.564051:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830007.586549:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830008.610476:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830009.634523:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830010.274373:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274380:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830010.274383:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274389:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274397:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274404:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787830010.274440:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78abb80 x1874663333439744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274468:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274472:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830010.274474:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274478:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274483:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274487:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274495:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830010.274507:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333439872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274524:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274527:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830010.274529:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274533:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274536:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274540:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274545:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830010.274559:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333440000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830010.274559:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333439744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1787830010.274582:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274585:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830010.274587:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274591:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274594:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274598:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274603:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830010.274617:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333440128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274632:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274635:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830010.274637:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274641:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274645:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274649:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274653:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830010.274663:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333440256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274671:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274674:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830010.274676:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274680:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274683:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274686:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274690:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830010.274698:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333440384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274706:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274708:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830010.274710:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274713:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274716:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274719:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274722:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830010.274729:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333440512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830010.274737:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830010.274739:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830010.274741:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830010.274744:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11821 00000100:00080000:1.0:1787830010.274746:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830010.274751:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1787830010.274754:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1787830010.274757:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830010.274768:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333440640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830010.274830:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78abb80 x1874663333439744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.274849:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78abb80 x1874663333439744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.274860:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.274864:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.274868:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78abb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333439744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787830010.274877:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830010.274918:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.274932:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78aa680 x1874663333439872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.274943:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333439872/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.274951:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.274954:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.274957:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333439872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830010.274980:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830010.275000:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275015:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78a9c00 x1874663333440000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275026:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333440000/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275033:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275036:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275039:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830010.275045:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830010.275078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275089:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78aa300 x1874663333440128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275100:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333440128/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275111:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830010.275120:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830010.275138:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275148:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78a8a80 x1874663333440256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275159:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333440256/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275169:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275171:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830010.275177:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830010.275196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275206:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78a9180 x1874663333440384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275217:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333440384/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275318:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275321:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830010.275328:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830010.275349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275360:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78a8700 x1874663333440512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275372:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333440512/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275383:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275386:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830010.275391:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830010.275409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830010.275419:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830010/real 1787830010] req@ffff9057c78a9500 x1874663333440640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830010.275429:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333440640/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830010.275443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830010.275446:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830010.275450:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333440640:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787830010.658455:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830011.682580:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830012.706469:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830013.730570:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830014.754480:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830015.394346:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394356:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830015.394359:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394369:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394379:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394387:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787830015.394423:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78ab100 x1874663333440768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394458:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394462:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830015.394464:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394469:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394472:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394476:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394485:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830015.394500:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9500 x1874663333440896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394510:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394513:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830015.394514:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394518:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394521:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394524:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394528:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830015.394538:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8700 x1874663333441024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394547:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394550:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830015.394552:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394555:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394558:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394561:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394566:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830015.394576:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9180 x1874663333441152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394584:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394586:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830015.394588:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394591:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394594:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394597:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394601:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830015.394612:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a8a80 x1874663333441280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394620:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394622:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830015.394624:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394628:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394630:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394634:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394639:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830015.394648:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa300 x1874663333441408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394657:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394659:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830015.394661:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394664:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394668:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394672:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394676:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830015.394686:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78a9c00 x1874663333441536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830015.394695:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830015.394697:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830015.394699:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830015.394703:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11826 00000100:00080000:1.0:1787830015.394705:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830015.394708:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1787830015.394713:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830015.394723:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057c78aa680 x1874663333441664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830015.394800:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333440768:192.168.203.156@tcp:250:kworker.0 00000800:00000100:2.0:1787830015.394857:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.394895:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78ab100 x1874663333440768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.394912:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78ab100 x1874663333440768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.394921:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.394924:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.394928:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78ab100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333440768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787830015.394934:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830015.394956:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.394967:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78a9500 x1874663333440896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.394979:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9500 x1874663333440896/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.394986:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.394990:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.394992:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333440896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830015.394997:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830015.395015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395025:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78a8700 x1874663333441024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395036:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8700 x1874663333441024/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395043:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395046:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395049:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830015.395055:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830015.395074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395085:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78a9180 x1874663333441152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395096:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9180 x1874663333441152/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395103:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395106:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395109:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830015.395115:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830015.395132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395142:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78a8a80 x1874663333441280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395153:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a8a80 x1874663333441280/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395165:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395168:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830015.395173:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830015.395191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395201:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78aa300 x1874663333441408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395212:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa300 x1874663333441408/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395248:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395251:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830015.395256:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830015.395274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395285:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78a9c00 x1874663333441536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395304:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78a9c00 x1874663333441536/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395313:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395316:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78a9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830015.395322:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830015.395339:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830015.395349:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830015/real 1787830015] req@ffff9057c78aa680 x1874663333441664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830015.395359:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057c78aa680 x1874663333441664/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830015.395366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830015.395369:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830015.395372:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057c78aa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333441664:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787830015.778593:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830016.802842:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830017.826664:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830018.850423:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830019.874496:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830020.514517:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514527:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830020.514531:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514541:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514551:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514558:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787830020.514595:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221500 x1874663333441792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830020.514662:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333441792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787830020.514723:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514727:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830020.514729:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514734:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514738:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514743:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514751:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830020.514765:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220a80 x1874663333441920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514775:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514778:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830020.514780:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514783:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514786:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514789:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514794:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830020.514803:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223b80 x1874663333442048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514812:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514815:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830020.514816:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514820:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514823:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514826:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514830:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830020.514840:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221180 x1874663333442176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514848:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514851:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830020.514852:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514856:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514859:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514862:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514866:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830020.514874:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222680 x1874663333442304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514882:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514885:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830020.514887:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514891:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514894:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514898:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514902:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830020.514911:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220e00 x1874663333442432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514919:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514922:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830020.514924:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514927:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514930:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514933:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514937:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830020.514947:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223800 x1874663333442560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830020.514956:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830020.514959:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830020.514960:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830020.514963:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11831 00000100:00080000:3.0:1787830020.514966:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830020.514969:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830020.514973:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830020.514983:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220000 x1874663333442688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830020.515044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515080:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb221500 x1874663333441792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515096:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221500 x1874663333441792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515110:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515114:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333441792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787830020.515135:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830020.515157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515169:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb220a80 x1874663333441920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515181:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220a80 x1874663333441920/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515192:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515195:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333441920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830020.515201:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830020.515245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515258:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb223b80 x1874663333442048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515270:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223b80 x1874663333442048/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515332:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515336:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830020.515342:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830020.515364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515376:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb221180 x1874663333442176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515388:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221180 x1874663333442176/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515399:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515402:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830020.515408:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830020.515428:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515439:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb222680 x1874663333442304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515450:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222680 x1874663333442304/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515456:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515460:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515462:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1787830020.515468:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830020.515486:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515496:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb220e00 x1874663333442432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515507:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220e00 x1874663333442432/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515515:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515518:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515522:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830020.515527:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1787830020.515545:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515555:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb223800 x1874663333442560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515566:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223800 x1874663333442560/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515576:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515579:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1787830020.515584:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1787830020.515602:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1787830020.515613:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830020/real 1787830020] req@ffff9057fb220000 x1874663333442688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830020.515624:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220000 x1874663333442688/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830020.515637:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830020.515640:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1787830020.515643:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333442688:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787830020.898407:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830021.924552:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830022.946488:0:49149:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830023.970599:0:6634:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830024.994401:0:49150:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830025.635480:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635503:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830025.635516:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635529:0:229536:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635542:0:229536:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635551:0:229536:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1787830025.635597:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1c00 x1874663333442816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635647:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635651:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830025.635653:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635658:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635662:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635667:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635705:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830025.635707:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333442816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787830025.635722:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1f80 x1874663333442944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635734:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635739:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.635741:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635745:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635749:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635754:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635759:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830025.635771:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0e00 x1874663333443072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635781:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635785:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830025.635787:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635791:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635795:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635801:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1787830025.635802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635809:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830025.635823:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0000 x1874663333443200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635832:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635836:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.635838:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635842:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635846:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635850:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635857:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830025.635863:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de1c00 x1874663333442816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830025.635870:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de1500 x1874663333443328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635885:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635889:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830025.635891:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830025.635895:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1c00 x1874663333442816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830025.635898:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635904:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635912:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635920:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830025.635924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.635931:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.635937:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333442816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1787830025.635937:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de0700 x1874663333443456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635953:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.635957:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.635959:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.635964:0:229536:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.635968:0:229536:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.635971:0:229536:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.635976:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830025.635986:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de2a00 x1874663333443584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.635997:0:229536:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.636000:0:229536:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.636002:0:229536:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.636006:0:229536:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11836 00000100:00080000:2.0:1787830025.636009:0:229536:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.636012:0:229536:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1787830025.636016:0:229536:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830025.636026:0:229536:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057d0de3100 x1874663333443712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830025.636452:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830025.636483:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.636498:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de1f80 x1874663333442944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.636511:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1f80 x1874663333442944/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.636519:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.636523:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.636526:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333442944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830025.636533:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830025.636553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.636564:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de0e00 x1874663333443072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.636575:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0e00 x1874663333443072/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.636583:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.636586:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.636773:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830025.636781:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830025.636806:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.636818:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de0000 x1874663333443200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.637187:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0000 x1874663333443200/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.637196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.637200:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.637203:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830025.637210:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830025.637252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.637264:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de1500 x1874663333443328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.637276:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de1500 x1874663333443328/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.637283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.637287:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.637290:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830025.637296:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830025.637524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.637536:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de0700 x1874663333443456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.637548:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de0700 x1874663333443456/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.637557:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.637560:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.637564:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830025.637570:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830025.637776:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.637791:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de2a00 x1874663333443584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.637802:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de2a00 x1874663333443584/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.637809:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.637812:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.637815:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830025.637821:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830025.637849:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830025.637858:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff9057d0de3100 x1874663333443712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.637868:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057d0de3100 x1874663333443712/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.637880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.637883:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830025.637894:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057d0de3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333443712:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1787830026.018404:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830027.042466:0:6166:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830028.066409:0:34879:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830029.090385:0:15672:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830030.114756:0:15673:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830030.754351:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fd94cc3d-2daf-4fbe-98ba-f53926e75993->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.754361:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830030.754366:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754375:0:149405:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754385:0:149405:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754392:0:149405:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff9057c5565800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1787830030.754428:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222d80 x1874663333443840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754451:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.754455:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830030.754456:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754460:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754464:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754468:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057d128a800: import ffff9057c752b800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754477:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830030.754489:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb223100 x1874663333443968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754500:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830030.754502:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333443840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787830030.754505:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.754509:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754514:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754517:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754521:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057d128a800: import ffff9057c7538800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754526:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830030.754539:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220380 x1874663333444096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754555:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.754558:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830030.754560:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754563:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11841 00000800:00000100:1.0:1787830030.754569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787830030.754569:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754576:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9057d128a800: import ffff9057c753c800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754582:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830030.754594:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222300 x1874663333444224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754603:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) 398e0287-d3c7-44dd-b1d9-6b85f698e615->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.754606:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.754608:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830030.754612:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb222d80 x1874663333443840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830030.754615:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754622:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754629:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057d128a800: import ffff9057c753a000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754634:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830030.754646:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222d80 x1874663333443840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830030.754659:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb220700 x1874663333444352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830030.754660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c5565800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.754669:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787830030.754675:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830030.754677:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fd94cc3d-2daf-4fbe-98ba-f53926e75993:2392:1874663333443840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1787830030.754683:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830030.754686:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754692:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754696:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.754700:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9057c7539000: import ffff9057c83db000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1787830030.754706:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443968:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1787830030.754706:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830030.754723:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb222a00 x1874663333444480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754733:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830030.754737:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1787830030.754740:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.754743:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.754749:0:149405:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754753:0:149405:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830030.754756:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb223100 x1874663333443968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830030.754760:0:149405:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9057c7539000: import ffff9057c83da800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754768:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830030.754775:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb223100 x1874663333443968/t0(0) o38->lustre-MDT0000-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.754785:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c752b800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830030.754789:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221c00 x1874663333444608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.754799:0:149405:0:(pinger.c:229:ptlrpc_pinger_process_import()) fc0b407b-a541-420d-8712-b9c8da18fe8c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830030.754803:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1787830030.754805:0:149405:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.754808:0:149405:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830030.754812:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb223100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333443968:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787830030.754816:0:149405:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: connect to NID 192.168.203.156@tcp last attempt 11841 00000100:00080000:3.0:1787830030.754821:0:149405:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830030.754824:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1787830030.754827:0:149405:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9057c7539000: import ffff9057c83d9800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1787830030.754834:0:149405:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830030.754846:0:149405:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9057fb221880 x1874663333444736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830030.754854:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.754868:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb220380 x1874663333444096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.754889:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220380 x1874663333444096/t0(0) o38->lustre-MDT0001-mdc-ffff9057d128a800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.754898:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c7538800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.754902:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.754906:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830030.754924:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830030.754954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.754964:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb222300 x1874663333444224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.754976:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222300 x1874663333444224/t0(0) o8->lustre-OST0000-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.754983:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.754986:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.754988:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830030.754994:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830030.755016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.755026:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb220700 x1874663333444352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.755045:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb220700 x1874663333444352/t0(0) o8->lustre-OST0001-osc-ffff9057d128a800@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.755052:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c753a000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.755055:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.755058:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb220700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:398e0287-d3c7-44dd-b1d9-6b85f698e615:2392:1874663333444352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1787830030.755074:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830030.755094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.755104:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb222a00 x1874663333444480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.755115:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb222a00 x1874663333444480/t0(0) o38->lustre-MDT0000-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.755124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83db000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.755127:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.755130:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb222a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830030.755135:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1787830030.755152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.755161:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb221c00 x1874663333444608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.755172:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221c00 x1874663333444608/t0(0) o38->lustre-MDT0001-mdc-ffff9057c7539000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.755180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83da800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.755183:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.755186:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1787830030.755191:0:2392:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1787830030.755208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1787830030.755217:0:2392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff9057fb221880 x1874663333444736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.755258:0:2392:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9057fb221880 x1874663333444736/t0(0) o8->lustre-OST0001-osc-ffff9057c7539000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.755266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff9057c83d9800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.755269:0:2392:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1787830030.755271:0:2392:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9057fb221880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc0b407b-a541-420d-8712-b9c8da18fe8c:2392:1874663333444736:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1787830031.138445:0:34880:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830032.162476:0:34881:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 74072 lines, 74072 kept, 0 dropped, 0 bad.